2024-08-09T14:31:47.167 INFO:root:teuthology version: 0.0.1.dev425+g43b8805 2024-08-09T14:31:47.187 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-09T14:31:47.264 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi/7845532 branch: wip-yuri7-testing-2024-08-08-0804-squid description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7845532' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri7-testing-2024-08-08-0804-squid ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 0a88632b45f25ae38493667c7dbd155f6e59b499 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 0a88632b45f25ae38493667c7dbd155f6e59b499 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri7-testing-2024-08-08-0804-squid sha1: 0a88632b45f25ae38493667c7dbd155f6e59b499 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 2176 sha1: 0a88632b45f25ae38493667c7dbd155f6e59b499 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri7-testing-2024-08-08-0804-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0a88632b45f25ae38493667c7dbd155f6e59b499 targets: smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK8ZDYPXlgeQwzp6qxSwh8yTH3mHyBIaCZdSUcRsZDoojfhkT3rTAm4rHXr/Ye+KbmUkOKWA/0/Dm9LwuzWhXDA= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKrU4g/VdCy1BITchK7Ie7Ofan1QAueLNbN3w8++rDVpZHWFa1fNDxQp3zRr//NeAnyV2FUIGBAlCdgi8LZZPUg= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 43b88056bcf6a0548bdd2d0502e56290cbae52df timestamp: 2024-08-09_13:31:16 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-09T14:31:47.264 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa; will attempt to use it 2024-08-09T14:31:47.265 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks 2024-08-09T14:31:47.265 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-09T14:31:47.267 INFO:teuthology.task.internal:Checking packages... 2024-08-09T14:31:47.288 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '0a88632b45f25ae38493667c7dbd155f6e59b499' 2024-08-09T14:31:47.288 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-09T14:31:47.288 INFO:teuthology.packaging:ref: None 2024-08-09T14:31:47.288 INFO:teuthology.packaging:tag: None 2024-08-09T14:31:47.288 INFO:teuthology.packaging:branch: wip-yuri7-testing-2024-08-08-0804-squid 2024-08-09T14:31:47.288 INFO:teuthology.packaging:sha1: 0a88632b45f25ae38493667c7dbd155f6e59b499 2024-08-09T14:31:47.288 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri7-testing-2024-08-08-0804-squid 2024-08-09T14:31:47.435 INFO:teuthology.task.internal:Found packages for ceph version 19.1.0-1345.g0a88632b 2024-08-09T14:31:47.436 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-09T14:31:47.444 INFO:teuthology.task.internal:no buildpackages task found 2024-08-09T14:31:47.444 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-09T14:31:47.455 INFO:teuthology.task.internal:Saving configuration 2024-08-09T14:31:47.469 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-09T14:31:47.478 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-09T14:31:47.505 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi/7845532', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-09 14:26:28.913708', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK8ZDYPXlgeQwzp6qxSwh8yTH3mHyBIaCZdSUcRsZDoojfhkT3rTAm4rHXr/Ye+KbmUkOKWA/0/Dm9LwuzWhXDA='} 2024-08-09T14:31:47.524 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi/7845532', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-09 14:26:28.912095', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKrU4g/VdCy1BITchK7Ie7Ofan1QAueLNbN3w8++rDVpZHWFa1fNDxQp3zRr//NeAnyV2FUIGBAlCdgi8LZZPUg='} 2024-08-09T14:31:47.524 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-09T14:31:47.532 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-08-09T14:31:47.532 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-08-09T14:31:47.532 INFO:teuthology.run_tasks:Running task console_log... 2024-08-09T14:31:47.591 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9d7d18bbe0>, signals=[15]) 2024-08-09T14:31:47.591 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-09T14:31:47.599 INFO:teuthology.task.internal:Opening connections... 2024-08-09T14:31:47.599 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-08-09T14:31:47.600 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-09T14:31:47.675 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-08-09T14:31:47.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-09T14:31:47.752 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-09T14:31:47.760 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-08-09T14:31:47.777 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-08-09T14:31:47.777 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-08-09T14:31:47.834 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Stream" 2024-08-09T14:31:47.838 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="9" 2024-08-09T14:31:47.838 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2024-08-09T14:31:47.838 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2024-08-09T14:31:47.838 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="9" 2024-08-09T14:31:47.838 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el9" 2024-08-09T14:31:47.838 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-09T14:31:47.838 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2024-08-09T14:31:47.838 INFO:teuthology.orchestra.run.smithi155.stdout:LOGO="fedora-logo-icon" 2024-08-09T14:31:47.838 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-09T14:31:47.839 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://centos.org/" 2024-08-09T14:31:47.839 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-09T14:31:47.839 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-09T14:31:47.839 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-09T14:31:47.840 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-08-09T14:31:47.855 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-08-09T14:31:47.873 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-08-09T14:31:47.874 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-08-09T14:31:47.930 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-08-09T14:31:47.931 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="9" 2024-08-09T14:31:47.931 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-08-09T14:31:47.931 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-08-09T14:31:47.931 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="9" 2024-08-09T14:31:47.931 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el9" 2024-08-09T14:31:47.931 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-09T14:31:47.931 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-08-09T14:31:47.931 INFO:teuthology.orchestra.run.smithi194.stdout:LOGO="fedora-logo-icon" 2024-08-09T14:31:47.931 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-09T14:31:47.931 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-08-09T14:31:47.931 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-09T14:31:47.932 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-09T14:31:47.932 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-09T14:31:47.932 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-08-09T14:31:47.964 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-09T14:31:47.974 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-09T14:31:47.981 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-09T14:31:47.981 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-08-09T14:31:47.983 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-08-09T14:31:47.999 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-09T14:31:48.006 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-09T14:31:48.006 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-08-09T14:31:48.040 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-08-09T14:31:48.147 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-09T14:31:48.203 INFO:teuthology.run_tasks:Running task kernel... 2024-08-09T14:31:48.218 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-09T14:31:48.218 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-09T14:31:48.219 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-09T14:31:48.219 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-09T14:31:48.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-09T14:31:48.236 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-08-09T14:31:48.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-09T14:31:48.237 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-08-09T14:31:48.294 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-437.el9.x86_64 2024-08-09T14:31:48.294 INFO:teuthology.task.kernel:Running kernel on smithi155: 5.14.0-437.el9.x86_64 2024-08-09T14:31:48.294 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum install -y kernel 2024-08-09T14:31:48.295 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-437.el9.x86_64 2024-08-09T14:31:48.295 INFO:teuthology.task.kernel:Running kernel on smithi194: 5.14.0-437.el9.x86_64 2024-08-09T14:31:48.295 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum install -y kernel 2024-08-09T14:31:51.421 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.2 MB 00:01 2024-08-09T14:31:51.531 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 5.3 MB/s | 8.2 MB 00:01 2024-08-09T14:31:54.672 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 20 MB 00:02 2024-08-09T14:31:59.073 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.6 MB 00:00 2024-08-09T14:32:00.740 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 43 kB/s | 18 kB 00:00 2024-08-09T14:32:01.577 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 31 MB/s | 22 MB 00:00 2024-08-09T14:32:06.866 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-08-09T14:32:08.881 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-09T14:32:08.881 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-09T14:32:08.980 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:32:08.992 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:32:08.992 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-08-09T14:32:08.993 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:32:08.993 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:32:08.993 INFO:teuthology.orchestra.run.smithi194.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-09T14:32:08.993 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-08-09T14:32:08.993 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-09T14:32:08.993 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-09T14:32:08.993 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-09T14:32:08.993 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:32:08.993 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:32:08.993 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:32:08.993 INFO:teuthology.orchestra.run.smithi194.stdout:Install 4 Packages 2024-08-09T14:32:08.993 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:32:08.994 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 84 M 2024-08-09T14:32:08.994 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 126 M 2024-08-09T14:32:08.994 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:32:09.569 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 20 MB 00:16 2024-08-09T14:32:09.576 INFO:teuthology.orchestra.run.smithi194.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 4.9 MB/s | 1.5 MB 00:00 2024-08-09T14:32:14.686 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.6 MB 00:01 2024-08-09T14:32:16.850 INFO:teuthology.orchestra.run.smithi194.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 2.3 MB/s | 17 MB 00:07 2024-08-09T14:32:17.078 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 18 kB 00:01 2024-08-09T14:32:18.107 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 25 MB/s | 22 MB 00:00 2024-08-09T14:32:23.433 INFO:teuthology.orchestra.run.smithi194.stdout:(3/4): kernel-modules-core-5.14.0-480.el9.x86_6 2.2 MB/s | 30 MB 00:13 2024-08-09T14:32:23.463 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-08-09T14:32:25.435 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-09T14:32:25.436 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-09T14:32:25.531 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:32:25.539 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:32:25.539 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-08-09T14:32:25.539 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:32:25.539 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:32:25.539 INFO:teuthology.orchestra.run.smithi155.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-09T14:32:25.539 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-08-09T14:32:25.539 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-09T14:32:25.539 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-09T14:32:25.539 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-09T14:32:25.539 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:32:25.540 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:32:25.540 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:32:25.540 INFO:teuthology.orchestra.run.smithi155.stdout:Install 4 Packages 2024-08-09T14:32:25.540 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:32:25.540 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 84 M 2024-08-09T14:32:25.540 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 126 M 2024-08-09T14:32:25.541 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:32:25.650 INFO:teuthology.orchestra.run.smithi194.stdout:(4/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 2.2 MB/s | 36 MB 00:16 2024-08-09T14:32:25.650 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:32:25.650 INFO:teuthology.orchestra.run.smithi194.stdout:Total 5.1 MB/s | 84 MB 00:16 2024-08-09T14:32:26.385 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:32:26.735 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:32:26.735 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:32:26.930 INFO:teuthology.orchestra.run.smithi155.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 1.4 MB/s | 1.5 MB 00:01 2024-08-09T14:32:27.372 INFO:teuthology.orchestra.run.smithi155.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 11 MB/s | 17 MB 00:01 2024-08-09T14:32:27.733 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:32:27.734 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:32:28.089 INFO:teuthology.orchestra.run.smithi155.stdout:(3/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 16 MB/s | 36 MB 00:02 2024-08-09T14:32:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:32:30.189 INFO:teuthology.orchestra.run.smithi155.stdout:(4/4): kernel-modules-core-5.14.0-480.el9.x86_6 9.2 MB/s | 30 MB 00:03 2024-08-09T14:32:30.189 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:32:30.190 INFO:teuthology.orchestra.run.smithi155.stdout:Total 18 MB/s | 84 MB 00:04 2024-08-09T14:32:30.828 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-09T14:32:30.923 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:32:30.924 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-09T14:32:31.295 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:32:31.295 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:32:32.326 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:32:32.327 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:32:33.586 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-09T14:32:33.732 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-09T14:32:34.785 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:32:35.532 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-09T14:32:35.630 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-09T14:32:38.297 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-09T14:32:38.446 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-09T14:32:38.566 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-09T14:32:38.707 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-09T14:32:43.278 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-09T14:32:43.396 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-09T14:32:43.573 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-09T14:32:48.275 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-09T14:33:13.146 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-09T14:33:13.186 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-09T14:33:13.536 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-09T14:33:13.536 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-09T14:33:13.536 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-09T14:33:13.536 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-09T14:33:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-09T14:33:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:33:14.339 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:33:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-09T14:33:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-09T14:33:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-09T14:33:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-09T14:33:14.339 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:33:14.339 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:33:14.724 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-08-09T14:33:14.809 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-09T14:33:14.809 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-09T14:33:14.809 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-09T14:33:14.810 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-09T14:33:14.810 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-09T14:33:14.810 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-08-09T14:33:14.810 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi194.front.sepia.ceph.com, path=None, version=distro) 2024-08-09T14:33:14.810 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-08-09T14:33:14.867 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-09T14:33:14.868 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-09T14:33:14.868 DEBUG:teuthology.orchestra.run.smithi194:> sudo rpm -qi grub2-tools 2024-08-09T14:33:14.908 INFO:teuthology.orchestra.run.smithi194.stdout:Name : grub2-tools 2024-08-09T14:33:14.908 INFO:teuthology.orchestra.run.smithi194.stdout:Epoch : 1 2024-08-09T14:33:14.908 INFO:teuthology.orchestra.run.smithi194.stdout:Version : 2.06 2024-08-09T14:33:14.909 INFO:teuthology.orchestra.run.smithi194.stdout:Release : 68.el9 2024-08-09T14:33:14.909 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: x86_64 2024-08-09T14:33:14.909 INFO:teuthology.orchestra.run.smithi194.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-09T14:33:14.909 INFO:teuthology.orchestra.run.smithi194.stdout:Group : Unspecified 2024-08-09T14:33:14.909 INFO:teuthology.orchestra.run.smithi194.stdout:Size : 8264254 2024-08-09T14:33:14.909 INFO:teuthology.orchestra.run.smithi194.stdout:License : GPLv3+ 2024-08-09T14:33:14.909 INFO:teuthology.orchestra.run.smithi194.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-09T14:33:14.909 INFO:teuthology.orchestra.run.smithi194.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-09T14:33:14.909 INFO:teuthology.orchestra.run.smithi194.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-09T14:33:14.909 INFO:teuthology.orchestra.run.smithi194.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-09T14:33:14.909 INFO:teuthology.orchestra.run.smithi194.stdout:Packager : builder@centos.org 2024-08-09T14:33:14.910 INFO:teuthology.orchestra.run.smithi194.stdout:Vendor : CentOS 2024-08-09T14:33:14.910 INFO:teuthology.orchestra.run.smithi194.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-09T14:33:14.910 INFO:teuthology.orchestra.run.smithi194.stdout:Summary : Support tools for GRUB. 2024-08-09T14:33:14.910 INFO:teuthology.orchestra.run.smithi194.stdout:Description : 2024-08-09T14:33:14.910 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:33:14.910 INFO:teuthology.orchestra.run.smithi194.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-09T14:33:14.910 INFO:teuthology.orchestra.run.smithi194.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-09T14:33:14.910 INFO:teuthology.orchestra.run.smithi194.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-09T14:33:14.910 INFO:teuthology.orchestra.run.smithi194.stdout:hardware devices. 2024-08-09T14:33:14.910 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:33:14.910 INFO:teuthology.orchestra.run.smithi194.stdout:This subpackage provides tools for support of all platforms. 2024-08-09T14:33:14.911 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-09T14:33:14.911 INFO:teuthology.task.kernel:Updating grub on smithi194 to boot 5.14.0-480.el9.x86_64 2024-08-09T14:33:14.911 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-09T14:33:15.512 INFO:teuthology.orchestra.run.smithi194.stderr:Generating grub configuration file ... 2024-08-09T14:33:16.922 INFO:teuthology.orchestra.run.smithi194.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-09T14:33:16.954 INFO:teuthology.orchestra.run.smithi194.stderr:done 2024-08-09T14:33:16.957 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-08-09T14:33:16.971 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.727DMljzX0 2024-08-09T14:33:16.972 DEBUG:teuthology.orchestra.run.smithi194:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.727DMljzX0 2024-08-09T14:33:17.038 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.727DMljzX0 2024-08-09T14:33:17.178 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.727DMljzX0 is 6KB 2024-08-09T14:33:17.227 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.727DMljzX0 2024-08-09T14:33:17.241 DEBUG:teuthology.orchestra.run.smithi194:> sudo /bin/ls /boot/loader/entries || true 2024-08-09T14:33:17.329 INFO:teuthology.orchestra.run.smithi194.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-09T14:33:17.329 INFO:teuthology.orchestra.run.smithi194.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-09T14:33:17.330 INFO:teuthology.orchestra.run.smithi194.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-09T14:33:17.330 INFO:teuthology.orchestra.run.smithi194.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-09T14:33:17.331 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-09T14:33:17.444 DEBUG:teuthology.orchestra.run.smithi194:> sudo shutdown -r now 2024-08-09T14:33:18.035 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-09T14:33:18.076 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-09T14:33:18.361 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-09T14:33:18.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-09T14:33:18.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-09T14:33:18.361 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-09T14:33:19.192 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-09T14:33:19.192 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:33:19.193 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:33:19.193 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-09T14:33:19.193 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-09T14:33:19.193 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-09T14:33:19.193 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-09T14:33:19.193 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:33:19.194 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:33:19.595 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-08-09T14:33:19.677 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-09T14:33:19.678 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-09T14:33:19.678 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-09T14:33:19.678 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-09T14:33:19.678 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-09T14:33:19.678 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-08-09T14:33:19.678 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2024-08-09T14:33:19.678 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q kernel | sort -rV | head -n 1 2024-08-09T14:33:19.735 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-09T14:33:19.735 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-09T14:33:19.736 DEBUG:teuthology.orchestra.run.smithi155:> sudo rpm -qi grub2-tools 2024-08-09T14:33:19.765 INFO:teuthology.orchestra.run.smithi155.stdout:Name : grub2-tools 2024-08-09T14:33:19.765 INFO:teuthology.orchestra.run.smithi155.stdout:Epoch : 1 2024-08-09T14:33:19.765 INFO:teuthology.orchestra.run.smithi155.stdout:Version : 2.06 2024-08-09T14:33:19.765 INFO:teuthology.orchestra.run.smithi155.stdout:Release : 68.el9 2024-08-09T14:33:19.765 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: x86_64 2024-08-09T14:33:19.765 INFO:teuthology.orchestra.run.smithi155.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-09T14:33:19.765 INFO:teuthology.orchestra.run.smithi155.stdout:Group : Unspecified 2024-08-09T14:33:19.766 INFO:teuthology.orchestra.run.smithi155.stdout:Size : 8264254 2024-08-09T14:33:19.766 INFO:teuthology.orchestra.run.smithi155.stdout:License : GPLv3+ 2024-08-09T14:33:19.766 INFO:teuthology.orchestra.run.smithi155.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-09T14:33:19.766 INFO:teuthology.orchestra.run.smithi155.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-09T14:33:19.766 INFO:teuthology.orchestra.run.smithi155.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-09T14:33:19.766 INFO:teuthology.orchestra.run.smithi155.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-09T14:33:19.766 INFO:teuthology.orchestra.run.smithi155.stdout:Packager : builder@centos.org 2024-08-09T14:33:19.766 INFO:teuthology.orchestra.run.smithi155.stdout:Vendor : CentOS 2024-08-09T14:33:19.766 INFO:teuthology.orchestra.run.smithi155.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-09T14:33:19.766 INFO:teuthology.orchestra.run.smithi155.stdout:Summary : Support tools for GRUB. 2024-08-09T14:33:19.766 INFO:teuthology.orchestra.run.smithi155.stdout:Description : 2024-08-09T14:33:19.766 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:33:19.767 INFO:teuthology.orchestra.run.smithi155.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-09T14:33:19.767 INFO:teuthology.orchestra.run.smithi155.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-09T14:33:19.767 INFO:teuthology.orchestra.run.smithi155.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-09T14:33:19.767 INFO:teuthology.orchestra.run.smithi155.stdout:hardware devices. 2024-08-09T14:33:19.767 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:33:19.767 INFO:teuthology.orchestra.run.smithi155.stdout:This subpackage provides tools for support of all platforms. 2024-08-09T14:33:19.768 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-09T14:33:19.768 INFO:teuthology.task.kernel:Updating grub on smithi155 to boot 5.14.0-480.el9.x86_64 2024-08-09T14:33:19.768 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-09T14:33:20.338 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2024-08-09T14:33:21.755 INFO:teuthology.orchestra.run.smithi155.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-09T14:33:21.792 INFO:teuthology.orchestra.run.smithi155.stderr:done 2024-08-09T14:33:21.795 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-08-09T14:33:21.810 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.1S3Zmtrdmf 2024-08-09T14:33:21.811 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1S3Zmtrdmf 2024-08-09T14:33:21.877 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.1S3Zmtrdmf 2024-08-09T14:33:22.011 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.1S3Zmtrdmf is 6KB 2024-08-09T14:33:22.031 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.1S3Zmtrdmf 2024-08-09T14:33:22.045 DEBUG:teuthology.orchestra.run.smithi155:> sudo /bin/ls /boot/loader/entries || true 2024-08-09T14:33:22.106 INFO:teuthology.orchestra.run.smithi155.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-09T14:33:22.106 INFO:teuthology.orchestra.run.smithi155.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-09T14:33:22.106 INFO:teuthology.orchestra.run.smithi155.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-09T14:33:22.106 INFO:teuthology.orchestra.run.smithi155.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-09T14:33:22.108 DEBUG:teuthology.orchestra.run.smithi155:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-09T14:33:22.267 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2024-08-09T14:33:47.472 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-08-09T14:33:47.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-08-09T14:33:47.474 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-09T14:33:52.274 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-08-09T14:33:52.275 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-08-09T14:33:52.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-09T14:34:05.923 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-08-09T14:34:10.782 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-08-09T14:34:14.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-08-09T14:34:14.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-09T14:34:19.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-08-09T14:34:19.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-09T14:34:22.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-08-09T14:34:34.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-08-09T14:34:34.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-09T14:34:53.279 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-08-09T14:35:08.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-08-09T14:35:08.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-09T14:35:10.566 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-08-09T14:35:11.072 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-08-09T14:35:11.072 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-480.el9.x86_64"... 2024-08-09T14:35:11.072 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-08-09T14:35:11.087 INFO:teuthology.orchestra.run.smithi155.stdout:5.14.0-480.el9.x86_64 2024-08-09T14:35:11.087 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-09T14:35:11.087 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-09T14:35:11.087 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-09T14:35:12.088 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-09T14:35:12.088 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-09T14:35:12.201 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2024-08-09T14:35:12.251 DEBUG:teuthology.parallel:result is None 2024-08-09T14:35:14.929 DEBUG:teuthology.orchestra.remote:timed out 2024-08-09T14:35:26.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-08-09T14:35:26.931 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-09T14:35:27.481 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-08-09T14:35:27.861 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-08-09T14:35:27.861 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-480.el9.x86_64"... 2024-08-09T14:35:27.861 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-08-09T14:35:27.878 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-480.el9.x86_64 2024-08-09T14:35:27.878 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-09T14:35:27.878 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-09T14:35:27.878 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-09T14:35:28.879 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-08-09T14:35:28.880 DEBUG:teuthology.orchestra.run.smithi194:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-09T14:35:28.962 INFO:teuthology.orchestra.run.smithi194.stdout:ttyS1 2024-08-09T14:35:28.988 DEBUG:teuthology.parallel:result is None 2024-08-09T14:35:28.989 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-09T14:35:28.998 INFO:teuthology.task.internal:Creating test directory... 2024-08-09T14:35:28.998 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-09T14:35:29.001 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-09T14:35:29.020 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-09T14:35:29.029 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-09T14:35:29.043 INFO:teuthology.task.internal:Creating archive directory... 2024-08-09T14:35:29.043 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-09T14:35:29.064 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-09T14:35:29.163 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-09T14:35:29.171 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-09T14:35:29.172 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-09T14:35:29.174 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-09T14:35:29.204 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-09T14:35:29.210 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-09T14:35:29.215 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-09T14:35:29.222 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-09T14:35:29.223 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-09T14:35:29.233 INFO:teuthology.task.internal:Configuring sudo... 2024-08-09T14:35:29.233 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-09T14:35:29.260 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-09T14:35:29.291 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-09T14:35:29.300 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-09T14:35:29.300 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-09T14:35:29.328 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-09T14:35:29.348 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-09T14:35:29.423 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-09T14:35:29.474 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-09T14:35:29.558 DEBUG:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-09T14:35:29.597 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:35:29.597 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-09T14:35:29.662 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-09T14:35:29.707 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-09T14:35:29.761 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-09T14:35:29.842 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-09T14:35:29.889 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:35:29.889 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-09T14:35:29.956 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-08-09T14:35:29.958 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-08-09T14:35:30.010 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-09T14:35:30.045 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-09T14:35:30.479 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-09T14:35:30.490 INFO:teuthology.task.internal:Starting timer... 2024-08-09T14:35:30.490 INFO:teuthology.run_tasks:Running task pcp... 2024-08-09T14:35:30.546 INFO:teuthology.run_tasks:Running task selinux... 2024-08-09T14:35:30.556 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-08-09T14:35:30.557 DEBUG:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2024-08-09T14:35:30.642 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: 2024-08-09T14:35:30.644 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-08-09T14:35:30.705 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-08-09T14:35:30.707 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-09T14:35:30.707 DEBUG:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2024-08-09T14:35:30.734 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2024-08-09T14:35:30.734 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-08-09T14:35:30.756 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-08-09T14:35:30.757 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi155.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-08-09T14:35:30.757 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-09T14:35:30.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-09T14:35:30.801 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-09T14:35:30.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-09T14:35:30.830 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-09T14:35:30.830 DEBUG:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2024-08-09T14:35:30.878 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-08-09T14:35:30.905 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-09T14:35:30.914 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-08-09T14:35:30.917 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-08-09T14:35:31.029 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-09T14:35:31.039 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-09T14:35:31.040 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi155.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-09T14:40:04.260 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi155.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-08-09T14:40:04.261 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-08-09T14:40:04.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-09T14:40:04.339 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-08-09T14:40:04.425 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-08-09T14:40:04.426 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-08-09T14:40:04.426 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-09T14:40:04.505 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-08-09T14:40:04.581 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-08-09T14:40:04.582 INFO:teuthology.run_tasks:Running task clock... 2024-08-09T14:40:04.591 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-09T14:40:04.592 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-09T14:40:04.592 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-09T14:40:04.594 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-09T14:40:04.595 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-09T14:40:04.621 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-09T14:40:04.636 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-09T14:40:04.651 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-09T14:40:04.659 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2024-08-09T14:40:04.665 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-09T14:40:04.670 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2024-08-09T14:40:04.685 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-09T14:40:04.690 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-08-09T14:40:04.698 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-09T14:40:04.701 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-08-09T14:40:04.715 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-09T14:40:04.729 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-09T14:40:04.760 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-08-09T14:40:04.763 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-09T14:40:04.763 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-08-09T14:40:04.764 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-09T14:40:04.764 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-09T14:40:04.764 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-09T14:40:04.764 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-09T14:40:04.789 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-08-09T14:40:04.792 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-09T14:40:04.792 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-08-09T14:40:04.792 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-09T14:40:04.792 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-09T14:40:04.793 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-09T14:40:04.793 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-09T14:40:04.794 INFO:teuthology.run_tasks:Running task install... 2024-08-09T14:40:04.802 DEBUG:teuthology.task.install:project ceph 2024-08-09T14:40:04.802 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '0a88632b45f25ae38493667c7dbd155f6e59b499'}} 2024-08-09T14:40:04.802 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '0a88632b45f25ae38493667c7dbd155f6e59b499'} 2024-08-09T14:40:04.802 INFO:teuthology.task.install:Using flavor: default 2024-08-09T14:40:04.807 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-08-09T14:40:04.807 INFO:teuthology.task.install:extra packages: [] 2024-08-09T14:40:04.808 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0a88632b45f25ae38493667c7dbd155f6e59b499', 'tag': None, 'wait_for_package': False} 2024-08-09T14:40:04.808 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0a88632b45f25ae38493667c7dbd155f6e59b499 2024-08-09T14:40:04.810 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '0a88632b45f25ae38493667c7dbd155f6e59b499', 'tag': None, 'wait_for_package': False} 2024-08-09T14:40:04.811 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0a88632b45f25ae38493667c7dbd155f6e59b499 2024-08-09T14:40:04.958 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-08-0804-squid/0a88632b45f25ae38493667c7dbd155f6e59b499/centos/9/flavors/default/ 2024-08-09T14:40:04.958 INFO:teuthology.task.install.rpm:Package version is 19.1.0-1345.g0a88632b 2024-08-09T14:40:04.966 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-08-0804-squid/0a88632b45f25ae38493667c7dbd155f6e59b499/centos/9/flavors/default/ 2024-08-09T14:40:04.966 INFO:teuthology.task.install.rpm:Package version is 19.1.0-1345.g0a88632b 2024-08-09T14:40:05.071 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-08-0804-squid/0a88632b45f25ae38493667c7dbd155f6e59b499/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-08-0804-squid/0a88632b45f25ae38493667c7dbd155f6e59b499/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-08-0804-squid/0a88632b45f25ae38493667c7dbd155f6e59b499/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-09T14:40:05.072 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:40:05.072 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-09T14:40:05.077 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-08-0804-squid/0a88632b45f25ae38493667c7dbd155f6e59b499/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-08-0804-squid/0a88632b45f25ae38493667c7dbd155f6e59b499/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri7-testing-2024-08-08-0804-squid/0a88632b45f25ae38493667c7dbd155f6e59b499/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-09T14:40:05.077 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:40:05.077 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-09T14:40:05.097 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-08-09T14:40:05.097 DEBUG:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0a88632b45f25ae38493667c7dbd155f6e59b499/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-09T14:40:05.102 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-08-09T14:40:05.102 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/0a88632b45f25ae38493667c7dbd155f6e59b499/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-09T14:40:05.163 DEBUG:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-09T14:40:05.167 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-09T14:40:05.238 DEBUG:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-09T14:40:05.242 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-09T14:40:05.302 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2024-08-09T14:40:05.304 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum clean all 2024-08-09T14:40:05.306 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-08-09T14:40:05.307 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-08-09T14:40:05.531 INFO:teuthology.orchestra.run.smithi194.stdout:45 files removed 2024-08-09T14:40:05.534 INFO:teuthology.orchestra.run.smithi155.stdout:45 files removed 2024-08-09T14:40:05.558 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw 2024-08-09T14:40:05.561 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw 2024-08-09T14:40:06.161 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 258 kB/s | 84 kB 00:00 2024-08-09T14:40:06.170 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 252 kB/s | 84 kB 00:00 2024-08-09T14:40:06.486 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 43 kB/s | 12 kB 00:00 2024-08-09T14:40:06.554 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-08-09T14:40:06.820 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 6.5 kB/s | 1.9 kB 00:00 2024-08-09T14:40:06.888 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 6.6 kB/s | 1.9 kB 00:00 2024-08-09T14:40:07.545 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-08-09T14:40:08.571 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.2 MB 00:01 2024-08-09T14:40:10.954 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 8.6 MB/s | 20 MB 00:02 2024-08-09T14:40:11.675 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - AppStream 9.9 MB/s | 20 MB 00:02 2024-08-09T14:40:15.305 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.6 MB 00:00 2024-08-09T14:40:16.806 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 18 kB 00:00 2024-08-09T14:40:17.685 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 30 MB/s | 22 MB 00:00 2024-08-09T14:40:18.148 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.6 MB 00:02 2024-08-09T14:40:20.665 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 18 kB 00:01 2024-08-09T14:40:21.544 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 30 MB/s | 22 MB 00:00 2024-08-09T14:40:22.974 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-08-09T14:40:25.318 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:40:25.326 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:40:25.327 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-08-09T14:40:25.327 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:40:25.327 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:40:25.327 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 11 M 2024-08-09T14:40:25.327 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-08-09T14:40:25.327 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 3.4 M 2024-08-09T14:40:25.327 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 3.2 M 2024-08-09T14:40:25.327 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-08-09T14:40:25.327 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-09T14:40:25.327 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 5.4 M 2024-08-09T14:40:25.328 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 21 M 2024-08-09T14:40:25.328 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 25 k 2024-08-09T14:40:25.328 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-09T14:40:25.328 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-09T14:40:25.328 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-09T14:40:25.328 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 751 k 2024-08-09T14:40:25.328 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-09T14:40:25.328 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-09T14:40:25.328 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-09T14:40:25.328 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 506 k 2024-08-09T14:40:25.328 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-09T14:40:25.328 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 5.3 M 2024-08-09T14:40:25.329 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-09T14:40:25.329 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-09T14:40:25.329 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-09T14:40:25.329 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 45 k 2024-08-09T14:40:25.329 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 136 k 2024-08-09T14:40:25.329 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 188 k 2024-08-09T14:40:25.329 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-09T14:40:25.329 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 372 k 2024-08-09T14:40:25.329 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 349 k 2024-08-09T14:40:25.329 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 116 k 2024-08-09T14:40:25.329 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-09T14:40:25.329 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-09T14:40:25.330 INFO:teuthology.orchestra.run.smithi194.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-09T14:40:25.330 INFO:teuthology.orchestra.run.smithi194.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-09T14:40:25.330 INFO:teuthology.orchestra.run.smithi194.stdout:Installing weak dependencies: 2024-08-09T14:40:25.330 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-09T14:40:25.330 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:40:25.330 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:40:25.330 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:40:25.330 INFO:teuthology.orchestra.run.smithi194.stdout:Install 30 Packages 2024-08-09T14:40:25.330 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 2 Packages 2024-08-09T14:40:25.330 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:40:25.331 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 61 M 2024-08-09T14:40:25.331 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:40:26.034 INFO:teuthology.orchestra.run.smithi194.stdout:(1/32): ceph-base-19.1.0-1345.g0a88632b.el9.x86 11 MB/s | 5.4 MB 00:00 2024-08-09T14:40:26.109 INFO:teuthology.orchestra.run.smithi194.stdout:(2/32): ceph-selinux-19.1.0-1345.g0a88632b.el9. 337 kB/s | 25 kB 00:00 2024-08-09T14:40:26.201 INFO:teuthology.orchestra.run.smithi194.stdout:(3/32): libcephfs2-19.1.0-1345.g0a88632b.el9.x8 8.0 MB/s | 751 kB 00:00 2024-08-09T14:40:26.342 INFO:teuthology.orchestra.run.smithi194.stdout:(4/32): ceph-radosgw-19.1.0-1345.g0a88632b.el9. 13 MB/s | 11 MB 00:00 2024-08-09T14:40:26.368 INFO:teuthology.orchestra.run.smithi194.stdout:(5/32): libradosstriper1-19.1.0-1345.g0a88632b. 3.0 MB/s | 506 kB 00:00 2024-08-09T14:40:26.426 INFO:teuthology.orchestra.run.smithi194.stdout:(6/32): python3-ceph-argparse-19.1.0-1345.g0a88 776 kB/s | 45 kB 00:00 2024-08-09T14:40:26.501 INFO:teuthology.orchestra.run.smithi194.stdout:(7/32): python3-ceph-common-19.1.0-1345.g0a8863 1.8 MB/s | 136 kB 00:00 2024-08-09T14:40:26.610 INFO:teuthology.orchestra.run.smithi194.stdout:(8/32): librgw2-19.1.0-1345.g0a88632b.el9.x86_6 20 MB/s | 5.3 MB 00:00 2024-08-09T14:40:26.635 INFO:teuthology.orchestra.run.smithi194.stdout:(9/32): python3-cephfs-19.1.0-1345.g0a88632b.el 1.4 MB/s | 188 kB 00:00 2024-08-09T14:40:26.669 INFO:teuthology.orchestra.run.smithi194.stdout:(10/32): python3-rados-19.1.0-1345.g0a88632b.el 6.2 MB/s | 372 kB 00:00 2024-08-09T14:40:26.694 INFO:teuthology.orchestra.run.smithi194.stdout:(11/32): python3-rbd-19.1.0-1345.g0a88632b.el9. 5.8 MB/s | 349 kB 00:00 2024-08-09T14:40:26.720 INFO:teuthology.orchestra.run.smithi194.stdout:(12/32): python3-rgw-19.1.0-1345.g0a88632b.el9. 2.3 MB/s | 116 kB 00:00 2024-08-09T14:40:26.908 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-08-09T14:40:26.928 INFO:teuthology.orchestra.run.smithi194.stdout:(13/32): ceph-common-19.1.0-1345.g0a88632b.el9. 15 MB/s | 21 MB 00:01 2024-08-09T14:40:27.229 INFO:teuthology.orchestra.run.smithi194.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 200 kB/s | 107 kB 00:00 2024-08-09T14:40:27.254 INFO:teuthology.orchestra.run.smithi194.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 139 kB/s | 45 kB 00:00 2024-08-09T14:40:27.279 INFO:teuthology.orchestra.run.smithi194.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 286 kB/s | 160 kB 00:00 2024-08-09T14:40:27.596 INFO:teuthology.orchestra.run.smithi194.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 157 kB/s | 42 kB 00:00 2024-08-09T14:40:27.621 INFO:teuthology.orchestra.run.smithi194.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 796 kB/s | 292 kB 00:00 2024-08-09T14:40:27.647 INFO:teuthology.orchestra.run.smithi194.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.5 MB/s | 662 kB 00:00 2024-08-09T14:40:27.905 INFO:teuthology.orchestra.run.smithi194.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 127 kB/s | 36 kB 00:00 2024-08-09T14:40:27.955 INFO:teuthology.orchestra.run.smithi194.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 181 kB/s | 47 kB 00:00 2024-08-09T14:40:28.006 INFO:teuthology.orchestra.run.smithi194.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 540 kB/s | 221 kB 00:00 2024-08-09T14:40:28.065 INFO:teuthology.orchestra.run.smithi194.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.9 MB/s | 309 kB 00:00 2024-08-09T14:40:28.090 INFO:teuthology.orchestra.run.smithi194.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-08-09T14:40:28.115 INFO:teuthology.orchestra.run.smithi194.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 243 kB/s | 26 kB 00:00 2024-08-09T14:40:28.141 INFO:teuthology.orchestra.run.smithi194.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-08-09T14:40:28.174 INFO:teuthology.orchestra.run.smithi194.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 5.6 MB/s | 192 kB 00:00 2024-08-09T14:40:28.224 INFO:teuthology.orchestra.run.smithi194.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-08-09T14:40:28.274 INFO:teuthology.orchestra.run.smithi194.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-08-09T14:40:28.383 INFO:teuthology.orchestra.run.smithi194.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-08-09T14:40:28.567 INFO:teuthology.orchestra.run.smithi194.stdout:(31/32): librados2-19.1.0-1345.g0a88632b.el9.x8 10 MB/s | 3.4 MB 00:00 2024-08-09T14:40:28.625 INFO:teuthology.orchestra.run.smithi194.stdout:(32/32): librbd1-19.1.0-1345.g0a88632b.el9.x86_ 9.0 MB/s | 3.2 MB 00:00 2024-08-09T14:40:28.630 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:40:28.630 INFO:teuthology.orchestra.run.smithi194.stdout:Total 18 MB/s | 61 MB 00:03 2024-08-09T14:40:28.726 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:40:28.762 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:40:28.762 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:40:29.242 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:40:29.245 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:40:29.245 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-08-09T14:40:29.245 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:40:29.245 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:40:29.245 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 11 M 2024-08-09T14:40:29.246 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2024-08-09T14:40:29.246 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 3.4 M 2024-08-09T14:40:29.246 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 3.2 M 2024-08-09T14:40:29.246 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-08-09T14:40:29.246 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-08-09T14:40:29.246 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 5.4 M 2024-08-09T14:40:29.246 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 21 M 2024-08-09T14:40:29.246 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 25 k 2024-08-09T14:40:29.246 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-09T14:40:29.246 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-08-09T14:40:29.246 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-08-09T14:40:29.246 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 751 k 2024-08-09T14:40:29.247 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-09T14:40:29.247 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-09T14:40:29.247 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-09T14:40:29.247 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 506 k 2024-08-09T14:40:29.247 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-09T14:40:29.247 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 5.3 M 2024-08-09T14:40:29.247 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-09T14:40:29.247 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-09T14:40:29.247 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-08-09T14:40:29.247 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 45 k 2024-08-09T14:40:29.247 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 136 k 2024-08-09T14:40:29.247 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 188 k 2024-08-09T14:40:29.247 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-09T14:40:29.248 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 372 k 2024-08-09T14:40:29.248 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 349 k 2024-08-09T14:40:29.248 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 116 k 2024-08-09T14:40:29.248 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-08-09T14:40:29.248 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-08-09T14:40:29.248 INFO:teuthology.orchestra.run.smithi155.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-08-09T14:40:29.248 INFO:teuthology.orchestra.run.smithi155.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-09T14:40:29.248 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2024-08-09T14:40:29.248 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-08-09T14:40:29.248 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:40:29.248 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:40:29.248 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:40:29.248 INFO:teuthology.orchestra.run.smithi155.stdout:Install 30 Packages 2024-08-09T14:40:29.249 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 2 Packages 2024-08-09T14:40:29.249 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:40:29.249 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 61 M 2024-08-09T14:40:29.249 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:40:29.347 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:40:29.348 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:40:30.771 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:40:30.855 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-09T14:40:30.888 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-08-09T14:40:30.909 INFO:teuthology.orchestra.run.smithi155.stdout:(1/32): ceph-radosgw-19.1.0-1345.g0a88632b.el9. 8.6 MB/s | 11 MB 00:01 2024-08-09T14:40:30.913 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-08-09T14:40:30.928 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-09T14:40:30.948 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-09T14:40:30.976 INFO:teuthology.orchestra.run.smithi155.stdout:(2/32): ceph-base-19.1.0-1345.g0a88632b.el9.x86 4.2 MB/s | 5.4 MB 00:01 2024-08-09T14:40:31.001 INFO:teuthology.orchestra.run.smithi155.stdout:(3/32): ceph-selinux-19.1.0-1345.g0a88632b.el9. 274 kB/s | 25 kB 00:00 2024-08-09T14:40:31.135 INFO:teuthology.orchestra.run.smithi155.stdout:(4/32): libradosstriper1-19.1.0-1345.g0a88632b. 3.7 MB/s | 506 kB 00:00 2024-08-09T14:40:31.138 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-09T14:40:31.160 INFO:teuthology.orchestra.run.smithi155.stdout:(5/32): libcephfs2-19.1.0-1345.g0a88632b.el9.x8 4.0 MB/s | 751 kB 00:00 2024-08-09T14:40:31.194 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-09T14:40:31.261 INFO:teuthology.orchestra.run.smithi155.stdout:(6/32): python3-ceph-argparse-19.1.0-1345.g0a88 451 kB/s | 45 kB 00:00 2024-08-09T14:40:31.352 INFO:teuthology.orchestra.run.smithi155.stdout:(7/32): python3-ceph-common-19.1.0-1345.g0a8863 1.4 MB/s | 136 kB 00:00 2024-08-09T14:40:31.444 INFO:teuthology.orchestra.run.smithi155.stdout:(8/32): python3-cephfs-19.1.0-1345.g0a88632b.el 2.0 MB/s | 188 kB 00:00 2024-08-09T14:40:31.544 INFO:teuthology.orchestra.run.smithi155.stdout:(9/32): python3-rados-19.1.0-1345.g0a88632b.el9 3.6 MB/s | 372 kB 00:00 2024-08-09T14:40:31.586 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-08-09T14:40:31.645 INFO:teuthology.orchestra.run.smithi155.stdout:(10/32): python3-rbd-19.1.0-1345.g0a88632b.el9. 3.4 MB/s | 349 kB 00:00 2024-08-09T14:40:31.673 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:19.1.0-1345.g0a88632b.el9.x86_64 7/34 2024-08-09T14:40:31.754 INFO:teuthology.orchestra.run.smithi155.stdout:(11/32): librgw2-19.1.0-1345.g0a88632b.el9.x86_ 8.6 MB/s | 5.3 MB 00:00 2024-08-09T14:40:31.770 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:19.1.0-1345.g0a88632b.el9.x86_64 7/34 2024-08-09T14:40:31.779 INFO:teuthology.orchestra.run.smithi155.stdout:(12/32): python3-rgw-19.1.0-1345.g0a88632b.el9. 870 kB/s | 116 kB 00:00 2024-08-09T14:40:31.865 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:19.1.0-1345.g0a88632b.el9.x86_64 8/34 2024-08-09T14:40:31.900 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:19.1.0-1345.g0a88632b.el9.x86_64 9/34 2024-08-09T14:40:31.963 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:19.1.0-1345.g0a88632b.el9.x86_64 9/34 2024-08-09T14:40:31.992 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-08-09T14:40:32.038 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:19.1.0-1345.g0a88632b.el9. 11/34 2024-08-09T14:40:32.071 INFO:teuthology.orchestra.run.smithi155.stdout:(13/32): ceph-common-19.1.0-1345.g0a88632b.el9. 8.9 MB/s | 21 MB 00:02 2024-08-09T14:40:32.096 INFO:teuthology.orchestra.run.smithi155.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 313 kB/s | 107 kB 00:00 2024-08-09T14:40:32.106 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:19.1.0-1345.g0a88632b.el9.x86_64 12/34 2024-08-09T14:40:32.121 INFO:teuthology.orchestra.run.smithi155.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 468 kB/s | 160 kB 00:00 2024-08-09T14:40:32.136 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:19.1.0-1345.g0a88632b.el9.x86_6 13/34 2024-08-09T14:40:32.192 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:19.1.0-1345.g0a88632b.el9.x86_6 13/34 2024-08-09T14:40:32.204 INFO:teuthology.orchestra.run.smithi155.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-08-09T14:40:32.227 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-08-09T14:40:32.230 INFO:teuthology.orchestra.run.smithi155.stdout:(17/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 285 kB/s | 45 kB 00:00 2024-08-09T14:40:32.264 INFO:teuthology.orchestra.run.smithi155.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.9 MB/s | 662 kB 00:00 2024-08-09T14:40:32.289 INFO:teuthology.orchestra.run.smithi155.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 503 kB/s | 42 kB 00:00 2024-08-09T14:40:32.314 INFO:teuthology.orchestra.run.smithi155.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 713 kB/s | 36 kB 00:00 2024-08-09T14:40:32.338 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-08-09T14:40:32.339 INFO:teuthology.orchestra.run.smithi155.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-08-09T14:40:32.370 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-08-09T14:40:32.390 INFO:teuthology.orchestra.run.smithi155.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 471 kB/s | 47 kB 00:00 2024-08-09T14:40:32.412 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-08-09T14:40:32.444 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-08-09T14:40:32.473 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-08-09T14:40:32.473 INFO:teuthology.orchestra.run.smithi155.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.9 MB/s | 309 kB 00:00 2024-08-09T14:40:32.499 INFO:teuthology.orchestra.run.smithi155.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-08-09T14:40:32.506 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-08-09T14:40:32.524 INFO:teuthology.orchestra.run.smithi155.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 242 kB/s | 26 kB 00:00 2024-08-09T14:40:32.549 INFO:teuthology.orchestra.run.smithi155.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-08-09T14:40:32.574 INFO:teuthology.orchestra.run.smithi155.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 7.6 MB/s | 192 kB 00:00 2024-08-09T14:40:32.617 INFO:teuthology.orchestra.run.smithi155.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-08-09T14:40:32.667 INFO:teuthology.orchestra.run.smithi155.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-08-09T14:40:32.751 INFO:teuthology.orchestra.run.smithi155.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-08-09T14:40:32.822 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-08-09T14:40:32.869 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:19.1.0-1345.g0a88632b.el9.x86_64 22/34 2024-08-09T14:40:32.964 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:19.1.0-1345.g0a88632b.el9.x86_64 22/34 2024-08-09T14:40:32.991 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:19.1.0-1345.g0a88632b.el9.x86_64 23/34 2024-08-09T14:40:33.036 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-09T14:40:33.384 INFO:teuthology.orchestra.run.smithi155.stdout:(31/32): librbd1-19.1.0-1345.g0a88632b.el9.x86_ 4.4 MB/s | 3.2 MB 00:00 2024-08-09T14:40:33.513 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-09T14:40:33.539 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:19.1.0-1345.g0a88632b.el9.x86_64 26/34 2024-08-09T14:40:33.611 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:19.1.0-1345.g0a88632b.el9.x86_64 26/34 2024-08-09T14:40:33.651 INFO:teuthology.orchestra.run.smithi155.stdout:(32/32): librados2-19.1.0-1345.g0a88632b.el9.x8 3.3 MB/s | 3.4 MB 00:01 2024-08-09T14:40:33.656 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:40:33.656 INFO:teuthology.orchestra.run.smithi155.stdout:Total 14 MB/s | 61 MB 00:04 2024-08-09T14:40:33.688 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:19.1.0-1345.g0a88632b.el9.x86_64 27/34 2024-08-09T14:40:33.716 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-common-2:19.1.0-1345.g0a88632b.el9.x8 28/34 2024-08-09T14:40:33.755 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:40:33.790 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:40:33.790 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:40:34.359 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:40:34.359 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:40:35.774 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:40:35.861 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-08-09T14:40:35.889 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-08-09T14:40:35.914 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-08-09T14:40:35.937 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-09T14:40:35.964 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-08-09T14:40:36.004 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.1.0-1345.g0a88632b.el9.x86_64 29/34 2024-08-09T14:40:36.038 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:19.1.0-1345.g0a88632b.el9.x86_64 29/34 2024-08-09T14:40:36.130 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-09T14:40:36.184 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-08-09T14:40:36.592 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-08-09T14:40:36.599 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.1.0-1345.g0a88632b.el9.x86_64 29/34 2024-08-09T14:40:36.650 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:19.1.0-1345.g0a88632b.el9.x86_64 30/34 2024-08-09T14:40:36.657 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:19.1.0-1345.g0a88632b.el9.x86_64 7/34 2024-08-09T14:40:36.739 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:19.1.0-1345.g0a88632b.el9.x86_64 30/34 2024-08-09T14:40:36.739 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-09T14:40:36.739 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-09T14:40:36.739 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:40:36.747 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:19.1.0-1345.g0a88632b.el9.x86_64 7/34 2024-08-09T14:40:36.776 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:19.1.0-1345.g0a88632b.el9.x86_64 31/34 2024-08-09T14:40:36.844 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:19.1.0-1345.g0a88632b.el9.x86_64 8/34 2024-08-09T14:40:36.867 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:19.1.0-1345.g0a88632b.el9.x86_64 9/34 2024-08-09T14:40:36.947 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:19.1.0-1345.g0a88632b.el9.x86_64 9/34 2024-08-09T14:40:36.984 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-08-09T14:40:37.030 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:19.1.0-1345.g0a88632b.el9. 11/34 2024-08-09T14:40:37.096 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:19.1.0-1345.g0a88632b.el9.x86_64 12/34 2024-08-09T14:40:37.119 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:19.1.0-1345.g0a88632b.el9.x86_6 13/34 2024-08-09T14:40:37.184 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:19.1.0-1345.g0a88632b.el9.x86_6 13/34 2024-08-09T14:40:37.218 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-08-09T14:40:37.327 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-08-09T14:40:37.361 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-08-09T14:40:37.402 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-08-09T14:40:37.434 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-08-09T14:40:37.463 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-08-09T14:40:37.496 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-08-09T14:40:37.808 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-08-09T14:40:37.850 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:19.1.0-1345.g0a88632b.el9.x86_64 22/34 2024-08-09T14:40:37.929 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:19.1.0-1345.g0a88632b.el9.x86_64 22/34 2024-08-09T14:40:37.955 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:19.1.0-1345.g0a88632b.el9.x86_64 23/34 2024-08-09T14:40:37.992 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-08-09T14:40:38.468 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-08-09T14:40:38.495 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:19.1.0-1345.g0a88632b.el9.x86_64 26/34 2024-08-09T14:40:38.566 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:19.1.0-1345.g0a88632b.el9.x86_64 26/34 2024-08-09T14:40:38.669 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:19.1.0-1345.g0a88632b.el9.x86_64 27/34 2024-08-09T14:40:38.697 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:19.1.0-1345.g0a88632b.el9.x8 28/34 2024-08-09T14:40:40.962 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.1.0-1345.g0a88632b.el9.x86_64 29/34 2024-08-09T14:40:41.002 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:19.1.0-1345.g0a88632b.el9.x86_64 29/34 2024-08-09T14:40:41.581 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:19.1.0-1345.g0a88632b.el9.x86_64 29/34 2024-08-09T14:40:41.622 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:19.1.0-1345.g0a88632b.el9.x86_64 30/34 2024-08-09T14:40:41.705 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:19.1.0-1345.g0a88632b.el9.x86_64 30/34 2024-08-09T14:40:41.705 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-09T14:40:41.705 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-09T14:40:41.705 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:40:41.732 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:19.1.0-1345.g0a88632b.el9.x86_64 31/34 2024-08-09T14:40:46.552 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:19.1.0-1345.g0a88632b.el9.x86_64 31/34 2024-08-09T14:40:46.553 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-08-09T14:40:46.553 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-08-09T14:40:46.553 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-08-09T14:40:46.553 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-08-09T14:40:46.553 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-08-09T14:40:46.553 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /root 2024-08-09T14:40:46.553 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-08-09T14:40:46.553 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:40:47.571 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:19.1.0-1345.g0a88632b.el9.x86_64 32/34 2024-08-09T14:40:47.596 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:19.1.0-1345.g0a88632b.el9.x86_64 32/34 2024-08-09T14:40:47.596 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:40:47.596 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-09T14:40:47.596 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-09T14:40:47.596 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-09T14:40:47.596 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:40:47.598 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-09T14:40:47.682 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-09T14:40:47.682 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-09T14:40:48.815 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-09T14:40:48.815 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:19.1.0-1345.g0a88632b.el9.x86_64 1/34 2024-08-09T14:40:48.815 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:19.1.0-1345.g0a88632b.el9.x86_64 2/34 2024-08-09T14:40:48.815 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:19.1.0-1345.g0a88632b.el9.x86_64 3/34 2024-08-09T14:40:48.815 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:19.1.0-1345.g0a88632b.el9.x86_64 4/34 2024-08-09T14:40:48.815 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:19.1.0-1345.g0a88632b.el9.x86_64 5/34 2024-08-09T14:40:48.815 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:19.1.0-1345.g0a88632b.el9.x86_6 6/34 2024-08-09T14:40:48.815 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:19.1.0-1345.g0a88632b.el9.x86_64 7/34 2024-08-09T14:40:48.815 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:19.1.0-1345.g0a88632b.el9. 8/34 2024-08-09T14:40:48.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-common-2:19.1.0-1345.g0a88632b.el9.x8 9/34 2024-08-09T14:40:48.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:19.1.0-1345.g0a88632b.el9.x86_64 10/34 2024-08-09T14:40:48.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:19.1.0-1345.g0a88632b.el9.x86_64 11/34 2024-08-09T14:40:48.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:19.1.0-1345.g0a88632b.el9.x86_64 12/34 2024-08-09T14:40:48.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:19.1.0-1345.g0a88632b.el9.x86_64 13/34 2024-08-09T14:40:48.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-09T14:40:48.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-09T14:40:48.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-09T14:40:48.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-09T14:40:48.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-09T14:40:48.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-09T14:40:48.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-09T14:40:48.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-09T14:40:48.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-09T14:40:48.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-09T14:40:48.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-09T14:40:48.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-09T14:40:48.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-09T14:40:48.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-09T14:40:48.818 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-09T14:40:48.818 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-09T14:40:48.818 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-09T14:40:48.818 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:19.1.0-1345.g0a88632b.el9.x86_64 31/34 2024-08-09T14:40:48.818 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-09T14:40:48.818 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:19.1.0-1345.g0a88632b.el9.x86_64 33/34 2024-08-09T14:40:49.698 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-09T14:40:49.698 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:40:49.699 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-08-09T14:40:49.699 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.699 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.699 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:40:49.699 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-09T14:40:49.699 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.699 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.699 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.699 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.699 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-09T14:40:49.699 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-09T14:40:49.699 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-09T14:40:49.700 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.700 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-09T14:40:49.700 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-09T14:40:49.700 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-09T14:40:49.700 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.700 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-09T14:40:49.700 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.700 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-09T14:40:49.700 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-09T14:40:49.700 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-09T14:40:49.700 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.700 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.701 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.701 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-09T14:40:49.701 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.701 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.701 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:49.701 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-09T14:40:49.701 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-09T14:40:49.701 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-09T14:40:49.701 INFO:teuthology.orchestra.run.smithi194.stdout: re2-1:20211101-3.el9.x86_64 2024-08-09T14:40:49.701 INFO:teuthology.orchestra.run.smithi194.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-09T14:40:49.701 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:40:49.702 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:40:50.047 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-test 2024-08-09T14:40:50.602 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:28 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:40:50.728 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:40:50.729 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:40:50.729 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-08-09T14:40:50.729 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:40:50.729 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:40:50.729 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 49 M 2024-08-09T14:40:50.729 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-08-09T14:40:50.729 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 163 k 2024-08-09T14:40:50.729 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-09T14:40:50.729 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-09T14:40:50.729 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:40:50.730 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:40:50.730 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:40:50.730 INFO:teuthology.orchestra.run.smithi194.stdout:Install 4 Packages 2024-08-09T14:40:50.730 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:40:50.730 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 49 M 2024-08-09T14:40:50.730 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 209 M 2024-08-09T14:40:50.731 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:40:51.027 INFO:teuthology.orchestra.run.smithi194.stdout:(1/4): libcephsqlite-19.1.0-1345.g0a88632b.el9. 636 kB/s | 163 kB 00:00 2024-08-09T14:40:51.308 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:19.1.0-1345.g0a88632b.el9.x86_64 31/34 2024-08-09T14:40:51.309 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2024-08-09T14:40:51.309 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2024-08-09T14:40:51.309 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2024-08-09T14:40:51.309 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2024-08-09T14:40:51.309 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2024-08-09T14:40:51.309 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /root 2024-08-09T14:40:51.309 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2024-08-09T14:40:51.309 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:40:51.369 INFO:teuthology.orchestra.run.smithi194.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 186 kB/s | 64 kB 00:00 2024-08-09T14:40:51.394 INFO:teuthology.orchestra.run.smithi194.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 487 kB/s | 304 kB 00:00 2024-08-09T14:40:52.327 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:19.1.0-1345.g0a88632b.el9.x86_64 32/34 2024-08-09T14:40:52.353 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:19.1.0-1345.g0a88632b.el9.x86_64 32/34 2024-08-09T14:40:52.353 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:40:52.354 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-09T14:40:52.354 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-09T14:40:52.354 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-09T14:40:52.354 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:40:52.355 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-09T14:40:52.402 INFO:teuthology.orchestra.run.smithi194.stdout:(4/4): ceph-test-19.1.0-1345.g0a88632b.el9.x86_ 30 MB/s | 49 MB 00:01 2024-08-09T14:40:52.404 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:40:52.405 INFO:teuthology.orchestra.run.smithi194.stdout:Total 29 MB/s | 49 MB 00:01 2024-08-09T14:40:52.432 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:40:52.452 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:40:52.452 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:40:52.463 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-08-09T14:40:52.464 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-09T14:40:52.826 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:40:52.826 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:40:53.352 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:40:53.437 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-09T14:40:53.476 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-09T14:40:53.504 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephsqlite-2:19.1.0-1345.g0a88632b.el9.x86_64 3/4 2024-08-09T14:40:53.737 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-08-09T14:40:53.737 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:19.1.0-1345.g0a88632b.el9.x86_64 1/34 2024-08-09T14:40:53.737 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:19.1.0-1345.g0a88632b.el9.x86_64 2/34 2024-08-09T14:40:53.737 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:19.1.0-1345.g0a88632b.el9.x86_64 3/34 2024-08-09T14:40:53.737 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:19.1.0-1345.g0a88632b.el9.x86_64 4/34 2024-08-09T14:40:53.737 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:19.1.0-1345.g0a88632b.el9.x86_64 5/34 2024-08-09T14:40:53.738 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:19.1.0-1345.g0a88632b.el9.x86_6 6/34 2024-08-09T14:40:53.738 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:19.1.0-1345.g0a88632b.el9.x86_64 7/34 2024-08-09T14:40:53.738 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:19.1.0-1345.g0a88632b.el9. 8/34 2024-08-09T14:40:53.738 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:19.1.0-1345.g0a88632b.el9.x8 9/34 2024-08-09T14:40:53.738 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:19.1.0-1345.g0a88632b.el9.x86_64 10/34 2024-08-09T14:40:53.738 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:19.1.0-1345.g0a88632b.el9.x86_64 11/34 2024-08-09T14:40:53.738 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:19.1.0-1345.g0a88632b.el9.x86_64 12/34 2024-08-09T14:40:53.738 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:19.1.0-1345.g0a88632b.el9.x86_64 13/34 2024-08-09T14:40:53.738 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-08-09T14:40:53.738 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-08-09T14:40:53.738 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-08-09T14:40:53.739 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-08-09T14:40:53.739 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-08-09T14:40:53.741 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-08-09T14:40:53.742 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-08-09T14:40:53.742 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-08-09T14:40:53.742 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-08-09T14:40:53.742 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-08-09T14:40:53.742 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-08-09T14:40:53.742 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-08-09T14:40:53.742 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-08-09T14:40:53.742 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-08-09T14:40:53.742 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-08-09T14:40:53.742 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-08-09T14:40:53.742 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-08-09T14:40:53.743 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:19.1.0-1345.g0a88632b.el9.x86_64 31/34 2024-08-09T14:40:53.743 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-08-09T14:40:53.743 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:19.1.0-1345.g0a88632b.el9.x86_64 33/34 2024-08-09T14:40:54.507 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-08-09T14:40:54.507 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:40:54.507 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2024-08-09T14:40:54.507 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.507 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.507 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:40:54.507 INFO:teuthology.orchestra.run.smithi155.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-08-09T14:40:54.507 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.507 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.508 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.508 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.508 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-09T14:40:54.508 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-9.0.0-11.el9.x86_64 2024-08-09T14:40:54.508 INFO:teuthology.orchestra.run.smithi155.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-08-09T14:40:54.508 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.508 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-09T14:40:54.508 INFO:teuthology.orchestra.run.smithi155.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-09T14:40:54.508 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-09T14:40:54.508 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.508 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-09T14:40:54.509 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.509 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-09T14:40:54.509 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-09T14:40:54.509 INFO:teuthology.orchestra.run.smithi155.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-08-09T14:40:54.509 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.509 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.509 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.509 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-09T14:40:54.509 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.509 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.509 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:40:54.509 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-24.02.0-1.el9.x86_64 2024-08-09T14:40:54.510 INFO:teuthology.orchestra.run.smithi155.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-08-09T14:40:54.510 INFO:teuthology.orchestra.run.smithi155.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-08-09T14:40:54.510 INFO:teuthology.orchestra.run.smithi155.stdout: re2-1:20211101-3.el9.x86_64 2024-08-09T14:40:54.510 INFO:teuthology.orchestra.run.smithi155.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-09T14:40:54.510 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:40:54.510 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:40:54.854 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-test 2024-08-09T14:40:55.401 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:29 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:40:55.527 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:40:55.528 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:40:55.528 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-08-09T14:40:55.528 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:40:55.528 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:40:55.528 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 49 M 2024-08-09T14:40:55.528 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-08-09T14:40:55.528 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 163 k 2024-08-09T14:40:55.528 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-09T14:40:55.528 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-09T14:40:55.529 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:40:55.529 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:40:55.529 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:40:55.529 INFO:teuthology.orchestra.run.smithi155.stdout:Install 4 Packages 2024-08-09T14:40:55.529 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:40:55.529 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 49 M 2024-08-09T14:40:55.530 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 209 M 2024-08-09T14:40:55.530 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:40:56.051 INFO:teuthology.orchestra.run.smithi155.stdout:(1/4): libcephsqlite-19.1.0-1345.g0a88632b.el9. 653 kB/s | 163 kB 00:00 2024-08-09T14:40:56.684 INFO:teuthology.orchestra.run.smithi155.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 344 kB/s | 304 kB 00:00 2024-08-09T14:40:56.710 INFO:teuthology.orchestra.run.smithi155.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 100 kB/s | 64 kB 00:00 2024-08-09T14:40:57.960 INFO:teuthology.orchestra.run.smithi155.stdout:(4/4): ceph-test-19.1.0-1345.g0a88632b.el9.x86_ 23 MB/s | 49 MB 00:02 2024-08-09T14:40:57.962 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:40:57.962 INFO:teuthology.orchestra.run.smithi155.stdout:Total 20 MB/s | 49 MB 00:02 2024-08-09T14:40:57.988 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:40:58.008 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:40:58.008 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:40:58.146 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephsqlite-2:19.1.0-1345.g0a88632b.el9.x86_64 3/4 2024-08-09T14:40:58.220 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:19.1.0-1345.g0a88632b.el9.x86_64 4/4 2024-08-09T14:40:58.376 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:40:58.376 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:40:58.916 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:40:58.998 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-09T14:40:59.042 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-09T14:40:59.070 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephsqlite-2:19.1.0-1345.g0a88632b.el9.x86_64 3/4 2024-08-09T14:41:00.473 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-test-2:19.1.0-1345.g0a88632b.el9.x86_64 4/4 2024-08-09T14:41:00.474 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:19.1.0-1345.g0a88632b.el9.x86_64 1/4 2024-08-09T14:41:00.474 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephsqlite-2:19.1.0-1345.g0a88632b.el9.x86_64 2/4 2024-08-09T14:41:00.474 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-09T14:41:00.761 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-09T14:41:00.761 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:00.762 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:41:00.762 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:00.762 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:00.762 INFO:teuthology.orchestra.run.smithi194.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-09T14:41:00.762 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-09T14:41:00.762 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:00.762 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:41:01.047 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph 2024-08-09T14:41:01.601 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:39 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:41:01.740 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:41:01.743 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:01.743 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-08-09T14:41:01.743 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:01.743 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:41:01.744 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 6.6 k 2024-08-09T14:41:01.744 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-08-09T14:41:01.744 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 2.4 M 2024-08-09T14:41:01.744 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 1.5 M 2024-08-09T14:41:01.744 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 249 k 2024-08-09T14:41:01.744 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 4.7 M 2024-08-09T14:41:01.744 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 17 M 2024-08-09T14:41:01.744 INFO:teuthology.orchestra.run.smithi194.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-09T14:41:01.744 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-09T14:41:01.744 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-09T14:41:01.744 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-09T14:41:01.745 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-09T14:41:01.745 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-09T14:41:01.745 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-09T14:41:01.745 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-09T14:41:01.745 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-09T14:41:01.745 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-09T14:41:01.745 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections 2024-08-09T14:41:01.745 INFO:teuthology.orchestra.run.smithi194.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-09T14:41:01.745 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-09T14:41:01.745 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-09T14:41:01.745 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-09T14:41:01.745 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-09T14:41:01.745 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-09T14:41:01.746 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-09T14:41:01.746 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-09T14:41:01.746 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-09T14:41:01.746 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-09T14:41:01.746 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-09T14:41:01.746 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-09T14:41:01.746 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-09T14:41:01.746 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-09T14:41:01.746 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-09T14:41:01.746 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-09T14:41:01.746 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-09T14:41:01.746 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-09T14:41:01.746 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-09T14:41:01.746 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-09T14:41:01.747 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-09T14:41:01.747 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:01.747 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:41:01.747 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:01.747 INFO:teuthology.orchestra.run.smithi194.stdout:Install 37 Packages 2024-08-09T14:41:01.747 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:01.747 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 30 M 2024-08-09T14:41:01.748 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 108 M 2024-08-09T14:41:01.748 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:41:02.821 INFO:teuthology.orchestra.run.smithi194.stdout:(1/37): ceph-19.1.0-1345.g0a88632b.el9.x86_64.r 34 kB/s | 6.6 kB 00:00 2024-08-09T14:41:03.013 INFO:teuthology.orchestra.run.smithi194.stdout:(2/37): ceph-mgr-19.1.0-1345.g0a88632b.el9.x86_ 3.9 MB/s | 1.5 MB 00:00 2024-08-09T14:41:03.096 INFO:teuthology.orchestra.run.smithi194.stdout:(3/37): ceph-mds-19.1.0-1345.g0a88632b.el9.x86_ 5.1 MB/s | 2.4 MB 00:00 2024-08-09T14:41:03.213 INFO:teuthology.orchestra.run.smithi194.stdout:(4/37): ceph-mgr-modules-core-19.1.0-1345.g0a88 2.1 MB/s | 249 kB 00:00 2024-08-09T14:41:03.405 INFO:teuthology.orchestra.run.smithi194.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-08-09T14:41:03.497 INFO:teuthology.orchestra.run.smithi194.stdout:(6/37): ceph-mon-19.1.0-1345.g0a88632b.el9.x86_ 7.0 MB/s | 4.7 MB 00:00 2024-08-09T14:41:03.631 INFO:teuthology.orchestra.run.smithi194.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 800 kB/s | 106 kB 00:00 2024-08-09T14:41:03.702 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephsqlite-2:19.1.0-1345.g0a88632b.el9.x86_64 3/4 2024-08-09T14:41:03.706 INFO:teuthology.orchestra.run.smithi194.stdout:(8/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.8 MB/s | 135 kB 00:00 2024-08-09T14:41:03.756 INFO:teuthology.orchestra.run.smithi194.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 3.5 MB/s | 1.2 MB 00:00 2024-08-09T14:41:03.776 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:19.1.0-1345.g0a88632b.el9.x86_64 4/4 2024-08-09T14:41:03.915 INFO:teuthology.orchestra.run.smithi194.stdout:(10/37): ceph-osd-19.1.0-1345.g0a88632b.el9.x86 19 MB/s | 17 MB 00:00 2024-08-09T14:41:03.940 INFO:teuthology.orchestra.run.smithi194.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 534 kB/s | 125 kB 00:00 2024-08-09T14:41:03.965 INFO:teuthology.orchestra.run.smithi194.stdout:(12/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-08-09T14:41:04.065 INFO:teuthology.orchestra.run.smithi194.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 347 kB/s | 35 kB 00:00 2024-08-09T14:41:04.099 INFO:teuthology.orchestra.run.smithi194.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-08-09T14:41:04.124 INFO:teuthology.orchestra.run.smithi194.stdout:(15/37): lua-5.4.4-4.el9.x86_64.rpm 903 kB/s | 188 kB 00:00 2024-08-09T14:41:04.149 INFO:teuthology.orchestra.run.smithi194.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-08-09T14:41:04.175 INFO:teuthology.orchestra.run.smithi194.stdout:(17/37): python3-packaging-20.9-5.el9.noarch.rp 706 kB/s | 77 kB 00:00 2024-08-09T14:41:04.200 INFO:teuthology.orchestra.run.smithi194.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-08-09T14:41:04.284 INFO:teuthology.orchestra.run.smithi194.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-08-09T14:41:04.317 INFO:teuthology.orchestra.run.smithi194.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 5.4 MB/s | 172 kB 00:00 2024-08-09T14:41:04.334 INFO:teuthology.orchestra.run.smithi194.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 121 kB/s | 22 kB 00:00 2024-08-09T14:41:04.351 INFO:teuthology.orchestra.run.smithi194.stdout:(22/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 288 kB/s | 43 kB 00:00 2024-08-09T14:41:04.376 INFO:teuthology.orchestra.run.smithi194.stdout:(23/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 254 kB/s | 11 kB 00:00 2024-08-09T14:41:04.426 INFO:teuthology.orchestra.run.smithi194.stdout:(24/37): python3-jaraco-collections-3.0.0-8.el9 901 kB/s | 23 kB 00:00 2024-08-09T14:41:04.452 INFO:teuthology.orchestra.run.smithi194.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 854 kB/s | 19 kB 00:00 2024-08-09T14:41:04.468 INFO:teuthology.orchestra.run.smithi194.stdout:(26/37): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2024-08-09T14:41:04.485 INFO:teuthology.orchestra.run.smithi194.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 585 kB/s | 20 kB 00:00 2024-08-09T14:41:04.502 INFO:teuthology.orchestra.run.smithi194.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-08-09T14:41:04.528 INFO:teuthology.orchestra.run.smithi194.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-08-09T14:41:04.554 INFO:teuthology.orchestra.run.smithi194.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-08-09T14:41:04.570 INFO:teuthology.orchestra.run.smithi194.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 388 kB/s | 16 kB 00:00 2024-08-09T14:41:04.595 INFO:teuthology.orchestra.run.smithi194.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-08-09T14:41:04.621 INFO:teuthology.orchestra.run.smithi194.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 715 kB/s | 36 kB 00:00 2024-08-09T14:41:04.688 INFO:teuthology.orchestra.run.smithi194.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-08-09T14:41:04.713 INFO:teuthology.orchestra.run.smithi194.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 806 kB/s | 20 kB 00:00 2024-08-09T14:41:04.738 INFO:teuthology.orchestra.run.smithi194.stdout:(36/37): python3-webob-1.8.7-6.el9.noarch.rpm 2.0 MB/s | 230 kB 00:00 2024-08-09T14:41:04.763 INFO:teuthology.orchestra.run.smithi194.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.0 MB/s | 427 kB 00:00 2024-08-09T14:41:04.767 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:04.767 INFO:teuthology.orchestra.run.smithi194.stdout:Total 10 MB/s | 30 MB 00:03 2024-08-09T14:41:04.913 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:41:04.953 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:41:04.953 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:41:05.403 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:41:05.403 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:41:06.175 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-test-2:19.1.0-1345.g0a88632b.el9.x86_64 4/4 2024-08-09T14:41:06.175 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:19.1.0-1345.g0a88632b.el9.x86_64 1/4 2024-08-09T14:41:06.175 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephsqlite-2:19.1.0-1345.g0a88632b.el9.x86_64 2/4 2024-08-09T14:41:06.175 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-09T14:41:06.459 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-09T14:41:06.459 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:06.459 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:41:06.459 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:06.459 INFO:teuthology.orchestra.run.smithi155.stdout: libcephsqlite-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:06.459 INFO:teuthology.orchestra.run.smithi155.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-09T14:41:06.459 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-09T14:41:06.459 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:06.460 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:41:06.704 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:41:06.737 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph 2024-08-09T14:41:06.766 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-09T14:41:06.798 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-09T14:41:06.826 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-09T14:41:06.897 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-09T14:41:06.959 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-09T14:41:06.992 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-09T14:41:07.083 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-09T14:41:07.170 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-09T14:41:07.211 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-08-09T14:41:07.246 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-09T14:41:07.284 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-09T14:41:07.303 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:41 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:41:07.318 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-09T14:41:07.352 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-09T14:41:07.392 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-09T14:41:07.435 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-09T14:41:07.440 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:41:07.443 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:41:07.443 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-08-09T14:41:07.443 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:41:07.443 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:41:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 6.6 k 2024-08-09T14:41:07.444 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-08-09T14:41:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 2.4 M 2024-08-09T14:41:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 1.5 M 2024-08-09T14:41:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 249 k 2024-08-09T14:41:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 4.7 M 2024-08-09T14:41:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 17 M 2024-08-09T14:41:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-08-09T14:41:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-08-09T14:41:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-08-09T14:41:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-09T14:41:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-09T14:41:07.444 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-09T14:41:07.445 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-09T14:41:07.445 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-09T14:41:07.445 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-09T14:41:07.445 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-09T14:41:07.445 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections 2024-08-09T14:41:07.445 INFO:teuthology.orchestra.run.smithi155.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-09T14:41:07.445 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-09T14:41:07.445 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-09T14:41:07.445 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-09T14:41:07.445 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-09T14:41:07.445 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-09T14:41:07.445 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-09T14:41:07.446 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-09T14:41:07.446 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-09T14:41:07.446 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-09T14:41:07.446 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-09T14:41:07.446 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-09T14:41:07.446 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-09T14:41:07.446 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-09T14:41:07.446 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-09T14:41:07.446 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-09T14:41:07.446 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-09T14:41:07.446 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-09T14:41:07.446 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-09T14:41:07.447 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-09T14:41:07.447 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-09T14:41:07.447 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:07.447 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:41:07.447 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:41:07.447 INFO:teuthology.orchestra.run.smithi155.stdout:Install 37 Packages 2024-08-09T14:41:07.447 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:07.448 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 30 M 2024-08-09T14:41:07.448 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 108 M 2024-08-09T14:41:07.448 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:41:07.470 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-09T14:41:07.518 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-09T14:41:07.550 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-09T14:41:07.631 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-09T14:41:07.769 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-09T14:41:07.850 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-09T14:41:07.903 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-09T14:41:07.942 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-09T14:41:07.991 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-09T14:41:08.045 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-09T14:41:08.094 INFO:teuthology.orchestra.run.smithi155.stdout:(1/37): ceph-19.1.0-1345.g0a88632b.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-08-09T14:41:08.240 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-09T14:41:08.287 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-09T14:41:08.365 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-09T14:41:08.453 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-09T14:41:08.486 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-09T14:41:08.610 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-09T14:41:08.669 INFO:teuthology.orchestra.run.smithi155.stdout:(2/37): ceph-mds-19.1.0-1345.g0a88632b.el9.x86_ 2.6 MB/s | 2.4 MB 00:00 2024-08-09T14:41:08.774 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-modules-core-2:19.1.0-1345.g0a88632b.el9. 32/37 2024-08-09T14:41:08.805 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:19.1.0-1345.g0a88632b.el9.x86_64 33/37 2024-08-09T14:41:08.832 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:19.1.0-1345.g0a88632b.el9.x86_64 33/37 2024-08-09T14:41:08.832 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:41:08.832 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-09T14:41:08.832 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-09T14:41:08.832 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-09T14:41:08.832 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:08.895 INFO:teuthology.orchestra.run.smithi155.stdout:(3/37): ceph-mgr-19.1.0-1345.g0a88632b.el9.x86_ 1.3 MB/s | 1.5 MB 00:01 2024-08-09T14:41:09.053 INFO:teuthology.orchestra.run.smithi155.stdout:(4/37): ceph-mgr-modules-core-19.1.0-1345.g0a88 1.5 MB/s | 249 kB 00:00 2024-08-09T14:41:09.829 INFO:teuthology.orchestra.run.smithi155.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 327 kB/s | 253 kB 00:00 2024-08-09T14:41:10.281 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:19.1.0-1345.g0a88632b.el9.x86_64 34/37 2024-08-09T14:41:10.308 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:19.1.0-1345.g0a88632b.el9.x86_64 34/37 2024-08-09T14:41:10.309 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:41:10.309 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-09T14:41:10.309 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-09T14:41:10.309 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-09T14:41:10.309 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:10.346 INFO:teuthology.orchestra.run.smithi155.stdout:(6/37): ceph-mon-19.1.0-1345.g0a88632b.el9.x86_ 2.1 MB/s | 4.7 MB 00:02 2024-08-09T14:41:10.740 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:19.1.0-1345.g0a88632b.el9.x86_64 35/37 2024-08-09T14:41:10.763 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:19.1.0-1345.g0a88632b.el9.x86_64 35/37 2024-08-09T14:41:10.764 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:41:10.764 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-09T14:41:10.764 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-09T14:41:10.764 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-09T14:41:10.764 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:10.829 INFO:teuthology.orchestra.run.smithi155.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 1.2 MB/s | 1.2 MB 00:00 2024-08-09T14:41:10.990 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:19.1.0-1345.g0a88632b.el9.x86_64 36/37 2024-08-09T14:41:10.996 INFO:teuthology.orchestra.run.smithi155.stdout:(8/37): ceph-osd-19.1.0-1345.g0a88632b.el9.x86_ 7.2 MB/s | 17 MB 00:02 2024-08-09T14:41:11.014 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:19.1.0-1345.g0a88632b.el9.x86_64 36/37 2024-08-09T14:41:11.014 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:41:11.014 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-09T14:41:11.015 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-09T14:41:11.015 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-09T14:41:11.015 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:11.079 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:19.1.0-1345.g0a88632b.el9.x86_64 37/37 2024-08-09T14:41:11.096 INFO:teuthology.orchestra.run.smithi155.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 142 kB/s | 106 kB 00:00 2024-08-09T14:41:11.682 INFO:teuthology.orchestra.run.smithi155.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 159 kB/s | 135 kB 00:00 2024-08-09T14:41:11.738 INFO:teuthology.orchestra.run.smithi155.stdout:(11/37): python3-requests-2.25.1-8.el9.noarch.r 168 kB/s | 125 kB 00:00 2024-08-09T14:41:11.839 INFO:teuthology.orchestra.run.smithi155.stdout:(12/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-08-09T14:41:11.864 INFO:teuthology.orchestra.run.smithi155.stdout:(13/37): python3-urllib3-1.26.5-6.el9.noarch.rp 280 kB/s | 215 kB 00:00 2024-08-09T14:41:11.889 INFO:teuthology.orchestra.run.smithi155.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 693 kB/s | 35 kB 00:00 2024-08-09T14:41:11.915 INFO:teuthology.orchestra.run.smithi155.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 981 kB/s | 172 kB 00:00 2024-08-09T14:41:11.940 INFO:teuthology.orchestra.run.smithi155.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-08-09T14:41:11.965 INFO:teuthology.orchestra.run.smithi155.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 831 kB/s | 42 kB 00:00 2024-08-09T14:41:11.990 INFO:teuthology.orchestra.run.smithi155.stdout:(18/37): lua-devel-5.4.4-4.el9.x86_64.rpm 444 kB/s | 22 kB 00:00 2024-08-09T14:41:12.015 INFO:teuthology.orchestra.run.smithi155.stdout:(19/37): python3-packaging-20.9-5.el9.noarch.rp 511 kB/s | 77 kB 00:00 2024-08-09T14:41:12.124 INFO:teuthology.orchestra.run.smithi155.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 953 kB/s | 151 kB 00:00 2024-08-09T14:41:12.149 INFO:teuthology.orchestra.run.smithi155.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 274 kB/s | 43 kB 00:00 2024-08-09T14:41:12.174 INFO:teuthology.orchestra.run.smithi155.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 474 kB/s | 11 kB 00:00 2024-08-09T14:41:12.200 INFO:teuthology.orchestra.run.smithi155.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 703 kB/s | 18 kB 00:00 2024-08-09T14:41:12.225 INFO:teuthology.orchestra.run.smithi155.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-08-09T14:41:12.250 INFO:teuthology.orchestra.run.smithi155.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-08-09T14:41:12.276 INFO:teuthology.orchestra.run.smithi155.stdout:(26/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-08-09T14:41:12.301 INFO:teuthology.orchestra.run.smithi155.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-08-09T14:41:12.326 INFO:teuthology.orchestra.run.smithi155.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 261 kB/s | 20 kB 00:00 2024-08-09T14:41:12.351 INFO:teuthology.orchestra.run.smithi155.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-08-09T14:41:12.376 INFO:teuthology.orchestra.run.smithi155.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-08-09T14:41:12.402 INFO:teuthology.orchestra.run.smithi155.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-08-09T14:41:12.427 INFO:teuthology.orchestra.run.smithi155.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-08-09T14:41:12.452 INFO:teuthology.orchestra.run.smithi155.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-08-09T14:41:12.486 INFO:teuthology.orchestra.run.smithi155.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-08-09T14:41:12.503 INFO:teuthology.orchestra.run.smithi155.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-08-09T14:41:12.528 INFO:teuthology.orchestra.run.smithi155.stdout:(36/37): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-08-09T14:41:12.553 INFO:teuthology.orchestra.run.smithi155.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-08-09T14:41:12.557 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:12.557 INFO:teuthology.orchestra.run.smithi155.stdout:Total 5.9 MB/s | 30 MB 00:05 2024-08-09T14:41:12.704 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:41:12.745 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-2:19.1.0-1345.g0a88632b.el9.x86_64 37/37 2024-08-09T14:41:12.745 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:19.1.0-1345.g0a88632b.el9.x86_64 1/37 2024-08-09T14:41:12.745 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:19.1.0-1345.g0a88632b.el9.x86_64 2/37 2024-08-09T14:41:12.745 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:19.1.0-1345.g0a88632b.el9.x86_64 3/37 2024-08-09T14:41:12.745 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:19.1.0-1345.g0a88632b.el9.x86_64 4/37 2024-08-09T14:41:12.745 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:19.1.0-1345.g0a88632b.el9.x86_64 5/37 2024-08-09T14:41:12.745 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-modules-core-2:19.1.0-1345.g0a88632b.el9. 6/37 2024-08-09T14:41:12.746 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-09T14:41:12.746 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-09T14:41:12.746 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-09T14:41:12.746 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-09T14:41:12.746 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-09T14:41:12.746 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-09T14:41:12.746 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-09T14:41:12.746 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-09T14:41:12.746 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-09T14:41:12.746 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-09T14:41:12.746 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-09T14:41:12.746 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-09T14:41:12.746 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-09T14:41:12.747 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-09T14:41:12.747 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-09T14:41:12.747 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-09T14:41:12.747 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-09T14:41:12.747 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-09T14:41:12.747 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-09T14:41:12.747 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-09T14:41:12.747 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-09T14:41:12.748 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:41:12.748 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:41:12.749 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-09T14:41:12.749 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-09T14:41:12.749 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-09T14:41:12.749 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-09T14:41:12.749 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-09T14:41:12.749 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-09T14:41:12.749 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-09T14:41:12.749 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-08-09T14:41:12.749 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-09T14:41:13.181 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:41:13.181 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:41:13.843 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-09T14:41:13.843 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:13.843 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:41:13.843 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:13.843 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:13.843 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:13.843 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:41:13.844 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:13.844 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:13.844 INFO:teuthology.orchestra.run.smithi194.stdout: lua-5.4.4-4.el9.x86_64 2024-08-09T14:41:13.844 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-09T14:41:13.844 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-09T14:41:13.844 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-09T14:41:13.844 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-09T14:41:13.844 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-09T14:41:13.844 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-09T14:41:13.844 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-09T14:41:13.844 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-09T14:41:13.845 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-09T14:41:13.845 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-09T14:41:13.845 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-09T14:41:13.845 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-09T14:41:13.845 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-09T14:41:13.845 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-09T14:41:13.845 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-09T14:41:13.845 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-09T14:41:13.845 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-09T14:41:13.845 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-09T14:41:13.845 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply-3.11-14.el9.noarch 2024-08-09T14:41:13.845 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-09T14:41:13.845 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-09T14:41:13.846 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-09T14:41:13.846 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-09T14:41:13.846 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-09T14:41:13.846 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-09T14:41:13.846 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-09T14:41:13.846 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-09T14:41:13.846 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-09T14:41:13.846 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-09T14:41:13.846 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-09T14:41:13.846 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:13.846 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:41:14.186 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-base 2024-08-09T14:41:14.478 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:41:14.539 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-08-09T14:41:14.572 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-08-09T14:41:14.600 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-08-09T14:41:14.673 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-08-09T14:41:14.716 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-08-09T14:41:14.737 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:52 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:41:14.752 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-08-09T14:41:14.822 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-base-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:41:14.841 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-08-09T14:41:14.863 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:41:14.865 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-08-09T14:41:14.865 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:41:14.906 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install cephadm 2024-08-09T14:41:14.927 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-08-09T14:41:14.967 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/37 2024-08-09T14:41:15.002 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-08-09T14:41:15.034 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-08-09T14:41:15.067 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-08-09T14:41:15.100 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-08-09T14:41:15.141 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-08-09T14:41:15.184 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-08-09T14:41:15.226 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-08-09T14:41:15.275 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-08-09T14:41:15.307 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-08-09T14:41:15.381 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-08-09T14:41:15.447 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:53 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:41:15.521 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-08-09T14:41:15.574 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:41:15.574 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:15.574 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-08-09T14:41:15.575 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:15.575 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:41:15.575 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 765 k 2024-08-09T14:41:15.575 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:15.575 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:41:15.575 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:15.575 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:41:15.575 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:15.575 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 765 k 2024-08-09T14:41:15.575 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 771 k 2024-08-09T14:41:15.575 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:41:15.607 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-08-09T14:41:15.659 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-08-09T14:41:15.698 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-08-09T14:41:15.747 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-08-09T14:41:15.802 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-08-09T14:41:15.999 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-08-09T14:41:16.061 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-08-09T14:41:16.139 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-08-09T14:41:16.228 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-08-09T14:41:16.268 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-08-09T14:41:16.298 INFO:teuthology.orchestra.run.smithi194.stdout:cephadm-19.1.0-1345.g0a88632b.el9.noarch.rpm 1.0 MB/s | 765 kB 00:00 2024-08-09T14:41:16.298 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:16.299 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.0 MB/s | 765 kB 00:00 2024-08-09T14:41:16.299 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:41:16.304 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:41:16.305 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:41:16.310 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:41:16.311 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:41:16.386 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:41:16.394 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-08-09T14:41:16.556 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:19.1.0-1345.g0a88632b.el9. 32/37 2024-08-09T14:41:16.586 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:19.1.0-1345.g0a88632b.el9.x86_64 33/37 2024-08-09T14:41:16.613 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:19.1.0-1345.g0a88632b.el9.x86_64 33/37 2024-08-09T14:41:16.613 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:41:16.613 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-09T14:41:16.613 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-09T14:41:16.613 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-09T14:41:16.614 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:16.851 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 1/1 2024-08-09T14:41:16.961 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 1/1 2024-08-09T14:41:17.327 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 1/1 2024-08-09T14:41:17.575 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 1/1 2024-08-09T14:41:17.575 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:17.575 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:41:17.575 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:41:17.575 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:17.575 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:41:17.769 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-immutable-object-cache 2024-08-09T14:41:18.062 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:19.1.0-1345.g0a88632b.el9.x86_64 34/37 2024-08-09T14:41:18.090 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:19.1.0-1345.g0a88632b.el9.x86_64 34/37 2024-08-09T14:41:18.091 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:41:18.091 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-09T14:41:18.091 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-09T14:41:18.091 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-09T14:41:18.091 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:18.330 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:56 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:41:18.459 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:41:18.460 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:18.460 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:41:18.460 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:18.461 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:41:18.461 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 145 k 2024-08-09T14:41:18.461 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:18.461 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:41:18.461 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:18.461 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:41:18.461 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:18.461 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 145 k 2024-08-09T14:41:18.461 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 435 k 2024-08-09T14:41:18.461 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:41:18.538 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:19.1.0-1345.g0a88632b.el9.x86_64 35/37 2024-08-09T14:41:18.562 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:19.1.0-1345.g0a88632b.el9.x86_64 35/37 2024-08-09T14:41:18.562 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:41:18.562 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-09T14:41:18.562 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-09T14:41:18.563 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-09T14:41:18.563 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:18.797 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:19.1.0-1345.g0a88632b.el9.x86_64 36/37 2024-08-09T14:41:18.821 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:19.1.0-1345.g0a88632b.el9.x86_64 36/37 2024-08-09T14:41:18.821 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:41:18.821 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-09T14:41:18.821 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-09T14:41:18.822 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-09T14:41:18.822 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:18.886 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:19.1.0-1345.g0a88632b.el9.x86_64 37/37 2024-08-09T14:41:18.951 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-immutable-object-cache-19.1.0-1345.g0a8863 296 kB/s | 145 kB 00:00 2024-08-09T14:41:18.951 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:18.951 INFO:teuthology.orchestra.run.smithi194.stdout:Total 296 kB/s | 145 kB 00:00 2024-08-09T14:41:18.952 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:41:18.959 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:41:18.960 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:41:19.017 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:41:19.017 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:41:19.161 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:41:19.268 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-2:19.1.0-1345.g0a88632b. 1/1 2024-08-09T14:41:19.295 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.0-1345.g0a88632b. 1/1 2024-08-09T14:41:19.295 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:41:19.295 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-09T14:41:19.295 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:20.206 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-2:19.1.0-1345.g0a88632b. 1/1 2024-08-09T14:41:20.206 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:20.206 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:41:20.207 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:20.207 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:20.207 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:41:20.423 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr 2024-08-09T14:41:20.514 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-2:19.1.0-1345.g0a88632b.el9.x86_64 37/37 2024-08-09T14:41:20.514 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:19.1.0-1345.g0a88632b.el9.x86_64 1/37 2024-08-09T14:41:20.514 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:19.1.0-1345.g0a88632b.el9.x86_64 2/37 2024-08-09T14:41:20.514 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:19.1.0-1345.g0a88632b.el9.x86_64 3/37 2024-08-09T14:41:20.514 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:19.1.0-1345.g0a88632b.el9.x86_64 4/37 2024-08-09T14:41:20.514 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:19.1.0-1345.g0a88632b.el9.x86_64 5/37 2024-08-09T14:41:20.514 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:19.1.0-1345.g0a88632b.el9. 6/37 2024-08-09T14:41:20.514 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-08-09T14:41:20.515 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-08-09T14:41:20.515 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-08-09T14:41:20.515 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-08-09T14:41:20.515 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-08-09T14:41:20.515 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-08-09T14:41:20.515 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-08-09T14:41:20.515 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-08-09T14:41:20.515 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-08-09T14:41:20.515 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-08-09T14:41:20.515 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-08-09T14:41:20.515 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-08-09T14:41:20.515 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-08-09T14:41:20.516 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-08-09T14:41:20.516 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-08-09T14:41:20.516 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-08-09T14:41:20.516 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-08-09T14:41:20.516 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-08-09T14:41:20.516 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-08-09T14:41:20.516 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-08-09T14:41:20.516 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-08-09T14:41:20.516 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-08-09T14:41:20.517 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-08-09T14:41:20.517 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-08-09T14:41:20.517 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-08-09T14:41:20.518 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-08-09T14:41:20.518 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-08-09T14:41:20.518 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-08-09T14:41:20.518 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 35/37 2024-08-09T14:41:20.518 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-08-09T14:41:20.985 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:58 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:41:21.073 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-mgr-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:41:21.117 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:41:21.118 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-08-09T14:41:21.118 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:41:21.189 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-dashboard 2024-08-09T14:41:21.492 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-08-09T14:41:21.492 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:21.492 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:41:21.492 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:21.492 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:21.492 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:21.492 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:41:21.493 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:21.493 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:21.493 INFO:teuthology.orchestra.run.smithi155.stdout: lua-5.4.4-4.el9.x86_64 2024-08-09T14:41:21.493 INFO:teuthology.orchestra.run.smithi155.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-08-09T14:41:21.493 INFO:teuthology.orchestra.run.smithi155.stdout: luarocks-3.9.2-5.el9.noarch 2024-08-09T14:41:21.493 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-09T14:41:21.493 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-09T14:41:21.493 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-09T14:41:21.493 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-09T14:41:21.493 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-09T14:41:21.493 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-09T14:41:21.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-09T14:41:21.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-09T14:41:21.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-09T14:41:21.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-09T14:41:21.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-09T14:41:21.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-09T14:41:21.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-09T14:41:21.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-09T14:41:21.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-09T14:41:21.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-09T14:41:21.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ply-3.11-14.el9.noarch 2024-08-09T14:41:21.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-09T14:41:21.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-09T14:41:21.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-09T14:41:21.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-09T14:41:21.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-09T14:41:21.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-09T14:41:21.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-09T14:41:21.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-09T14:41:21.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-09T14:41:21.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-09T14:41:21.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-09T14:41:21.495 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:21.496 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:41:21.734 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:59 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:41:21.819 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base 2024-08-09T14:41:21.862 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:41:21.863 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:21.863 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-08-09T14:41:21.863 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:21.863 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:41:21.863 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 3.6 M 2024-08-09T14:41:21.863 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-08-09T14:41:21.863 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-09T14:41:21.863 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 32 k 2024-08-09T14:41:21.863 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 16 k 2024-08-09T14:41:21.863 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-09T14:41:21.864 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-09T14:41:21.864 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-09T14:41:21.864 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-09T14:41:21.864 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-09T14:41:21.864 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-09T14:41:21.864 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-09T14:41:21.864 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-09T14:41:21.864 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:21.864 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:41:21.864 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:21.864 INFO:teuthology.orchestra.run.smithi194.stdout:Install 12 Packages 2024-08-09T14:41:21.864 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:21.865 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 8.7 M 2024-08-09T14:41:21.865 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 104 M 2024-08-09T14:41:21.865 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:41:22.368 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:56 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:41:22.452 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-base-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:41:22.494 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:41:22.495 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-08-09T14:41:22.495 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:41:22.536 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install cephadm 2024-08-09T14:41:22.582 INFO:teuthology.orchestra.run.smithi194.stdout:(1/12): ceph-grafana-dashboards-19.1.0-1345.g0a 87 kB/s | 32 kB 00:00 2024-08-09T14:41:22.615 INFO:teuthology.orchestra.run.smithi194.stdout:(2/12): ceph-prometheus-alerts-19.1.0-1345.g0a8 41 kB/s | 16 kB 00:00 2024-08-09T14:41:22.733 INFO:teuthology.orchestra.run.smithi194.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.3 MB/s | 269 kB 00:00 2024-08-09T14:41:22.808 INFO:teuthology.orchestra.run.smithi194.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-08-09T14:41:22.875 INFO:teuthology.orchestra.run.smithi194.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 8.0 MB/s | 548 kB 00:00 2024-08-09T14:41:22.900 INFO:teuthology.orchestra.run.smithi194.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 776 kB/s | 19 kB 00:00 2024-08-09T14:41:22.959 INFO:teuthology.orchestra.run.smithi194.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 35 MB/s | 2.0 MB 00:00 2024-08-09T14:41:23.000 INFO:teuthology.orchestra.run.smithi194.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.2 MB/s | 863 kB 00:00 2024-08-09T14:41:23.026 INFO:teuthology.orchestra.run.smithi194.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-08-09T14:41:23.076 INFO:teuthology.orchestra.run.smithi194.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.2 MB/s | 188 kB 00:00 2024-08-09T14:41:23.076 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:57 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:41:23.093 INFO:teuthology.orchestra.run.smithi194.stdout:(11/12): python3-repoze-lru-0.7-16.el9.noarch.r 334 kB/s | 31 kB 00:00 2024-08-09T14:41:23.151 INFO:teuthology.orchestra.run.smithi194.stdout:(12/12): ceph-mgr-dashboard-19.1.0-1345.g0a8863 3.8 MB/s | 3.6 MB 00:00 2024-08-09T14:41:23.155 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:23.155 INFO:teuthology.orchestra.run.smithi194.stdout:Total 6.7 MB/s | 8.7 MB 00:01 2024-08-09T14:41:23.202 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:41:23.202 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:41:23.203 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-08-09T14:41:23.203 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:41:23.203 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:41:23.203 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 765 k 2024-08-09T14:41:23.203 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:23.203 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:41:23.203 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:41:23.203 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:41:23.203 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:23.203 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 765 k 2024-08-09T14:41:23.203 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 771 k 2024-08-09T14:41:23.203 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:41:23.204 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:41:23.219 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:41:23.219 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:41:23.395 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:41:23.396 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:41:23.588 INFO:teuthology.orchestra.run.smithi155.stdout:cephadm-19.1.0-1345.g0a88632b.el9.noarch.rpm 1.9 MB/s | 765 kB 00:00 2024-08-09T14:41:23.589 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:23.589 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.9 MB/s | 765 kB 00:00 2024-08-09T14:41:23.589 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:41:23.594 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:41:23.594 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:41:23.600 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:41:23.600 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:41:23.670 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:41:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:41:24.002 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-08-09T14:41:24.038 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-08-09T14:41:24.084 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-08-09T14:41:24.114 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-08-09T14:41:24.149 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 1/1 2024-08-09T14:41:24.218 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-08-09T14:41:24.225 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 1/1 2024-08-09T14:41:24.356 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-08-09T14:41:24.458 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-08-09T14:41:24.502 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-08-09T14:41:24.530 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-08-09T14:41:24.562 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-prometheus-alerts-2:19.1.0-1345.g0a88632b.el9 10/12 2024-08-09T14:41:24.578 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 1/1 2024-08-09T14:41:24.823 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 1/1 2024-08-09T14:41:24.823 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:24.823 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:41:24.823 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:41:24.823 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:24.823 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:41:25.032 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-immutable-object-cache 2024-08-09T14:41:25.587 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:59 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:41:25.611 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-grafana-dashboards-2:19.1.0-1345.g0a88632b.el 11/12 2024-08-09T14:41:25.646 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-dashboard-2:19.1.0-1345.g0a88632b.el9.noa 12/12 2024-08-09T14:41:25.718 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:41:25.718 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:41:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:41:25.718 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:41:25.718 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:41:25.718 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 145 k 2024-08-09T14:41:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:25.719 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:41:25.719 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:41:25.719 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:41:25.719 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:25.720 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 145 k 2024-08-09T14:41:25.720 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 435 k 2024-08-09T14:41:25.720 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:41:26.136 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-immutable-object-cache-19.1.0-1345.g0a8863 349 kB/s | 145 kB 00:00 2024-08-09T14:41:26.136 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:26.137 INFO:teuthology.orchestra.run.smithi155.stdout:Total 347 kB/s | 145 kB 00:00 2024-08-09T14:41:26.137 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:41:26.144 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:41:26.144 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:41:26.200 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:41:26.201 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:41:26.346 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:41:26.436 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:19.1.0-1345.g0a88632b. 1/1 2024-08-09T14:41:26.460 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.0-1345.g0a88632b. 1/1 2024-08-09T14:41:26.460 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:41:26.460 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-09T14:41:26.460 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:26.974 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.0-1345.g0a88632b.el9.noa 12/12 2024-08-09T14:41:26.974 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-grafana-dashboards-2:19.1.0-1345.g0a88632b.el 1/12 2024-08-09T14:41:26.974 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-dashboard-2:19.1.0-1345.g0a88632b.el9.noa 2/12 2024-08-09T14:41:26.974 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-prometheus-alerts-2:19.1.0-1345.g0a88632b.el9 3/12 2024-08-09T14:41:26.974 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-08-09T14:41:26.974 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-08-09T14:41:26.974 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-08-09T14:41:26.974 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-08-09T14:41:26.975 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-08-09T14:41:26.975 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-08-09T14:41:26.975 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-08-09T14:41:26.975 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-08-09T14:41:27.299 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-08-09T14:41:27.299 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:27.299 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:41:27.299 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-09T14:41:27.299 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:41:27.299 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:41:27.299 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:41:27.299 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-09T14:41:27.299 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-09T14:41:27.300 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-09T14:41:27.300 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-09T14:41:27.300 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-09T14:41:27.300 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-09T14:41:27.300 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-09T14:41:27.300 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-09T14:41:27.300 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:27.300 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:41:27.344 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:19.1.0-1345.g0a88632b. 1/1 2024-08-09T14:41:27.344 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:27.344 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:41:27.344 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:27.345 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:27.345 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:41:27.548 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr 2024-08-09T14:41:27.552 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-09T14:41:28.101 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:06 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:41:28.105 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:02 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:41:28.193 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-mgr-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:41:28.228 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:41:28.229 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-08-09T14:41:28.229 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-08-09T14:41:28.229 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-08-09T14:41:28.229 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:41:28.230 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 7.4 M 2024-08-09T14:41:28.230 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-08-09T14:41:28.230 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-09T14:41:28.230 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-09T14:41:28.230 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-09T14:41:28.230 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-09T14:41:28.230 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-09T14:41:28.230 INFO:teuthology.orchestra.run.smithi194.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-09T14:41:28.230 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-09T14:41:28.230 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-09T14:41:28.230 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-09T14:41:28.231 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-09T14:41:28.231 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-09T14:41:28.231 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:28.231 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:41:28.231 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-08-09T14:41:28.231 INFO:teuthology.orchestra.run.smithi194.stdout:Install 12 Packages 2024-08-09T14:41:28.231 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:28.231 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 38 M 2024-08-09T14:41:28.232 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 209 M 2024-08-09T14:41:28.232 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:41:28.236 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:41:28.237 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-08-09T14:41:28.237 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:41:28.303 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-dashboard 2024-08-09T14:41:28.817 INFO:teuthology.orchestra.run.smithi194.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 988 kB/s | 195 kB 00:00 2024-08-09T14:41:28.854 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:02 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:41:28.908 INFO:teuthology.orchestra.run.smithi194.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.7 MB/s | 803 kB 00:00 2024-08-09T14:41:28.933 INFO:teuthology.orchestra.run.smithi194.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-08-09T14:41:28.982 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:41:28.983 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:41:28.983 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-08-09T14:41:28.983 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:41:28.984 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:41:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 3.6 M 2024-08-09T14:41:28.984 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-08-09T14:41:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-08-09T14:41:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 32 k 2024-08-09T14:41:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 16 k 2024-08-09T14:41:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-08-09T14:41:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-08-09T14:41:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-08-09T14:41:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-08-09T14:41:28.984 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-08-09T14:41:28.985 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-08-09T14:41:28.985 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-08-09T14:41:28.985 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-08-09T14:41:28.985 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:28.985 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:41:28.985 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:41:28.985 INFO:teuthology.orchestra.run.smithi155.stdout:Install 12 Packages 2024-08-09T14:41:28.985 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:28.986 INFO:teuthology.orchestra.run.smithi194.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-08-09T14:41:28.986 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 8.7 M 2024-08-09T14:41:28.986 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 104 M 2024-08-09T14:41:28.986 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:41:29.034 INFO:teuthology.orchestra.run.smithi194.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-08-09T14:41:29.243 INFO:teuthology.orchestra.run.smithi194.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.0 MB/s | 3.0 MB 00:00 2024-08-09T14:41:29.301 INFO:teuthology.orchestra.run.smithi194.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-08-09T14:41:29.385 INFO:teuthology.orchestra.run.smithi194.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-08-09T14:41:29.452 INFO:teuthology.orchestra.run.smithi194.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 3.7 MB/s | 250 kB 00:00 2024-08-09T14:41:29.594 INFO:teuthology.orchestra.run.smithi194.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-08-09T14:41:29.608 INFO:teuthology.orchestra.run.smithi155.stdout:(1/12): ceph-grafana-dashboards-19.1.0-1345.g0a 81 kB/s | 32 kB 00:00 2024-08-09T14:41:29.633 INFO:teuthology.orchestra.run.smithi155.stdout:(2/12): ceph-prometheus-alerts-19.1.0-1345.g0a8 39 kB/s | 16 kB 00:00 2024-08-09T14:41:29.994 INFO:teuthology.orchestra.run.smithi194.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 30 MB/s | 16 MB 00:00 2024-08-09T14:41:30.434 INFO:teuthology.orchestra.run.smithi155.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 337 kB/s | 269 kB 00:00 2024-08-09T14:41:30.592 INFO:teuthology.orchestra.run.smithi155.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:00 2024-08-09T14:41:30.651 INFO:teuthology.orchestra.run.smithi155.stdout:(5/12): ceph-mgr-dashboard-19.1.0-1345.g0a88632 2.5 MB/s | 3.6 MB 00:01 2024-08-09T14:41:30.653 INFO:teuthology.orchestra.run.smithi194.stdout:(12/12): ceph-mgr-diskprediction-local-19.1.0-1 3.6 MB/s | 7.4 MB 00:02 2024-08-09T14:41:30.654 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:30.654 INFO:teuthology.orchestra.run.smithi194.stdout:Total 16 MB/s | 38 MB 00:02 2024-08-09T14:41:30.734 INFO:teuthology.orchestra.run.smithi155.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 3.8 MB/s | 548 kB 00:00 2024-08-09T14:41:30.760 INFO:teuthology.orchestra.run.smithi155.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 179 kB/s | 19 kB 00:00 2024-08-09T14:41:30.793 INFO:teuthology.orchestra.run.smithi155.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.3 MB/s | 863 kB 00:00 2024-08-09T14:41:30.844 INFO:teuthology.orchestra.run.smithi155.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-08-09T14:41:30.869 INFO:teuthology.orchestra.run.smithi155.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 408 kB/s | 31 kB 00:00 2024-08-09T14:41:30.894 INFO:teuthology.orchestra.run.smithi155.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.8 MB/s | 188 kB 00:00 2024-08-09T14:41:30.944 INFO:teuthology.orchestra.run.smithi155.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.7 MB/s | 2.0 MB 00:00 2024-08-09T14:41:30.949 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:30.949 INFO:teuthology.orchestra.run.smithi155.stdout:Total 4.4 MB/s | 8.7 MB 00:01 2024-08-09T14:41:30.997 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:41:31.011 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:41:31.013 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:41:31.014 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:41:31.037 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:41:31.037 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:41:31.207 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:41:31.207 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:41:31.470 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:41:31.470 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:41:31.751 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:41:31.812 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-08-09T14:41:31.847 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-08-09T14:41:31.894 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-08-09T14:41:31.924 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-08-09T14:41:32.025 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-08-09T14:41:32.159 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-08-09T14:41:32.230 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:41:32.252 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-08-09T14:41:32.304 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-08-09T14:41:32.306 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-09T14:41:32.333 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-08-09T14:41:32.350 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-09T14:41:32.364 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-prometheus-alerts-2:19.1.0-1345.g0a88632b.el9 10/12 2024-08-09T14:41:32.449 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-09T14:41:32.493 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-09T14:41:32.669 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-09T14:41:32.700 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-09T14:41:32.776 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-09T14:41:33.421 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:19.1.0-1345.g0a88632b.el 11/12 2024-08-09T14:41:33.443 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-09T14:41:33.465 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:19.1.0-1345.g0a88632b.el9.noa 12/12 2024-08-09T14:41:33.546 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-09T14:41:34.776 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-09T14:41:34.875 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.0-1345.g0a88632b.el9.noa 12/12 2024-08-09T14:41:34.875 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:19.1.0-1345.g0a88632b.el 1/12 2024-08-09T14:41:34.875 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:19.1.0-1345.g0a88632b.el9.noa 2/12 2024-08-09T14:41:34.875 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-prometheus-alerts-2:19.1.0-1345.g0a88632b.el9 3/12 2024-08-09T14:41:34.875 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-08-09T14:41:34.876 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-08-09T14:41:34.876 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-08-09T14:41:34.876 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-08-09T14:41:34.876 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-08-09T14:41:34.876 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-08-09T14:41:34.876 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-08-09T14:41:34.876 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-08-09T14:41:35.189 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-08-09T14:41:35.189 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:35.189 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:41:35.189 INFO:teuthology.orchestra.run.smithi155.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-08-09T14:41:35.189 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:41:35.189 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:41:35.189 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-prometheus-alerts-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:41:35.190 INFO:teuthology.orchestra.run.smithi155.stdout: grpc-data-1.46.7-10.el9.noarch 2024-08-09T14:41:35.190 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-3.14.0-13.el9.x86_64 2024-08-09T14:41:35.190 INFO:teuthology.orchestra.run.smithi155.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-08-09T14:41:35.190 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-08-09T14:41:35.190 INFO:teuthology.orchestra.run.smithi155.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-08-09T14:41:35.190 INFO:teuthology.orchestra.run.smithi155.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-08-09T14:41:35.190 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-08-09T14:41:35.190 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.5.1-5.el9.noarch 2024-08-09T14:41:35.190 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:35.190 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:41:35.406 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-diskprediction-local 2024-08-09T14:41:35.801 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-09T14:41:35.832 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.0-1345.g0a886 12/12 2024-08-09T14:41:35.954 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:09 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:41:36.082 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:41:36.084 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-08-09T14:41:36.084 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-08-09T14:41:36.084 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-08-09T14:41:36.084 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:41:36.084 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 7.4 M 2024-08-09T14:41:36.084 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-08-09T14:41:36.084 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-08-09T14:41:36.084 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-08-09T14:41:36.084 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-08-09T14:41:36.084 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-08-09T14:41:36.084 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-08-09T14:41:36.084 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-08-09T14:41:36.085 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-08-09T14:41:36.085 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-08-09T14:41:36.085 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-08-09T14:41:36.085 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-08-09T14:41:36.085 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-08-09T14:41:36.085 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:36.085 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:41:36.085 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================= 2024-08-09T14:41:36.085 INFO:teuthology.orchestra.run.smithi155.stdout:Install 12 Packages 2024-08-09T14:41:36.085 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:41:36.086 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 38 M 2024-08-09T14:41:36.086 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 209 M 2024-08-09T14:41:36.086 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:41:37.095 INFO:teuthology.orchestra.run.smithi155.stdout:(1/12): ceph-mgr-diskprediction-local-19.1.0-13 16 MB/s | 7.4 MB 00:00 2024-08-09T14:41:37.137 INFO:teuthology.orchestra.run.smithi155.stdout:(2/12): libquadmath-11.4.1-3.el9.x86_64.rpm 389 kB/s | 195 kB 00:00 2024-08-09T14:41:37.354 INFO:teuthology.orchestra.run.smithi155.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 126 kB/s | 33 kB 00:00 2024-08-09T14:41:37.471 INFO:teuthology.orchestra.run.smithi155.stdout:(4/12): libgfortran-11.4.1-3.el9.x86_64.rpm 960 kB/s | 803 kB 00:00 2024-08-09T14:41:37.488 INFO:teuthology.orchestra.run.smithi155.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 136 kB/s | 18 kB 00:00 2024-08-09T14:41:37.580 INFO:teuthology.orchestra.run.smithi155.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 324 kB/s | 35 kB 00:00 2024-08-09T14:41:37.689 INFO:teuthology.orchestra.run.smithi155.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-08-09T14:41:37.780 INFO:teuthology.orchestra.run.smithi155.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.6 MB/s | 3.0 MB 00:00 2024-08-09T14:41:37.889 INFO:teuthology.orchestra.run.smithi155.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-08-09T14:41:38.357 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.0-1345.g0a886 12/12 2024-08-09T14:41:38.357 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.0-1345.g0a886 1/12 2024-08-09T14:41:38.357 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-09T14:41:38.358 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-09T14:41:38.358 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-09T14:41:38.358 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-09T14:41:38.358 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-09T14:41:38.358 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-09T14:41:38.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-09T14:41:38.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-09T14:41:38.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-09T14:41:38.359 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-09T14:41:38.573 INFO:teuthology.orchestra.run.smithi155.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 4.4 MB/s | 4.7 MB 00:01 2024-08-09T14:41:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-09T14:41:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:38.735 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:41:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:41:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-09T14:41:38.735 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-09T14:41:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-09T14:41:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-09T14:41:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-09T14:41:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-09T14:41:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-09T14:41:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-09T14:41:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-09T14:41:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-09T14:41:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-09T14:41:38.736 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:38.736 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:41:39.150 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-rook 2024-08-09T14:41:39.710 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:17 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:41:39.840 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:41:39.842 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:39.842 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-08-09T14:41:39.842 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:39.842 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:41:39.842 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 49 k 2024-08-09T14:41:39.842 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-08-09T14:41:39.842 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-09T14:41:39.842 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-09T14:41:39.843 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-08-09T14:41:39.843 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-09T14:41:39.843 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-09T14:41:39.843 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-09T14:41:39.843 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-09T14:41:39.843 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-09T14:41:39.843 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-09T14:41:39.843 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-09T14:41:39.843 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-09T14:41:39.843 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-09T14:41:39.843 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:39.843 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:41:39.844 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:39.844 INFO:teuthology.orchestra.run.smithi194.stdout:Install 13 Packages 2024-08-09T14:41:39.844 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:39.844 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 2.2 M 2024-08-09T14:41:39.845 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 26 M 2024-08-09T14:41:39.845 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:41:40.356 INFO:teuthology.orchestra.run.smithi155.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 1.9 MB/s | 5.1 MB 00:02 2024-08-09T14:41:40.491 INFO:teuthology.orchestra.run.smithi194.stdout:(1/13): ceph-mgr-rook-19.1.0-1345.g0a88632b.el9 228 kB/s | 49 kB 00:00 2024-08-09T14:41:40.649 INFO:teuthology.orchestra.run.smithi194.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 68 kB/s | 26 kB 00:00 2024-08-09T14:41:40.674 INFO:teuthology.orchestra.run.smithi194.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 48 kB/s | 19 kB 00:00 2024-08-09T14:41:40.891 INFO:teuthology.orchestra.run.smithi194.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 558 kB/s | 222 kB 00:00 2024-08-09T14:41:40.942 INFO:teuthology.orchestra.run.smithi194.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 543 kB/s | 159 kB 00:00 2024-08-09T14:41:41.000 INFO:teuthology.orchestra.run.smithi194.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 556 kB/s | 32 kB 00:00 2024-08-09T14:41:41.025 INFO:teuthology.orchestra.run.smithi194.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 795 kB/s | 279 kB 00:00 2024-08-09T14:41:41.050 INFO:teuthology.orchestra.run.smithi194.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 337 kB/s | 54 kB 00:00 2024-08-09T14:41:41.076 INFO:teuthology.orchestra.run.smithi194.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 187 kB/s | 14 kB 00:00 2024-08-09T14:41:41.160 INFO:teuthology.orchestra.run.smithi194.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.7 MB/s | 59 kB 00:00 2024-08-09T14:41:41.185 INFO:teuthology.orchestra.run.smithi194.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.9 MB/s | 90 kB 00:00 2024-08-09T14:41:41.218 INFO:teuthology.orchestra.run.smithi194.stdout:(12/13): python3-google-auth-2.32.0-1.el9.noarc 1.3 MB/s | 216 kB 00:00 2024-08-09T14:41:41.260 INFO:teuthology.orchestra.run.smithi194.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 4.9 MB/s | 1.0 MB 00:00 2024-08-09T14:41:41.264 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:41.265 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.6 MB/s | 2.2 MB 00:01 2024-08-09T14:41:41.327 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:41:41.342 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:41:41.342 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:41:41.429 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:41:41.431 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:41:41.935 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:41:42.109 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-09T14:41:42.169 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-09T14:41:42.233 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-09T14:41:42.271 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-09T14:41:42.312 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-09T14:41:42.417 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-09T14:41:42.524 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 7/13 2024-08-09T14:41:42.579 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-09T14:41:43.293 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-09T14:41:43.344 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-09T14:41:43.384 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-09T14:41:43.432 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-09T14:41:43.454 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-rook-2:19.1.0-1345.g0a88632b.el9.noarch 13/13 2024-08-09T14:41:44.209 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.0-1345.g0a88632b.el9.noarch 13/13 2024-08-09T14:41:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-rook-2:19.1.0-1345.g0a88632b.el9.noarch 1/13 2024-08-09T14:41:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-09T14:41:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-09T14:41:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-09T14:41:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-09T14:41:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-09T14:41:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-09T14:41:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-09T14:41:44.210 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-09T14:41:44.211 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 10/13 2024-08-09T14:41:44.211 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-09T14:41:44.211 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-09T14:41:44.536 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-09T14:41:44.537 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:44.537 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:41:44.537 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:41:44.537 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-09T14:41:44.537 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-09T14:41:44.537 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-08-09T14:41:44.537 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-09T14:41:44.537 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-09T14:41:44.537 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-09T14:41:44.537 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-09T14:41:44.537 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-09T14:41:44.537 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-09T14:41:44.538 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-09T14:41:44.538 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-09T14:41:44.538 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-09T14:41:44.538 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:44.538 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:41:44.794 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-cephadm 2024-08-09T14:41:45.340 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:23 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:41:45.469 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:41:45.474 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:45.474 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-08-09T14:41:45.474 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:45.475 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:41:45.475 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 146 k 2024-08-09T14:41:45.475 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-08-09T14:41:45.475 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-09T14:41:45.475 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-09T14:41:45.475 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-09T14:41:45.475 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-09T14:41:45.475 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-09T14:41:45.475 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:45.475 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:41:45.475 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:45.475 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-08-09T14:41:45.476 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:45.476 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 7.0 M 2024-08-09T14:41:45.476 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 33 M 2024-08-09T14:41:45.476 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:41:46.147 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): ceph-mgr-cephadm-19.1.0-1345.g0a88632b.e 712 kB/s | 146 kB 00:00 2024-08-09T14:41:46.173 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-08-09T14:41:46.256 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 693 kB/s | 58 kB 00:00 2024-08-09T14:41:46.281 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.0 MB/s | 548 kB 00:00 2024-08-09T14:41:46.307 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-08-09T14:41:47.324 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.3 MB/s | 6.0 MB 00:01 2024-08-09T14:41:47.326 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:47.326 INFO:teuthology.orchestra.run.smithi194.stdout:Total 3.8 MB/s | 7.0 MB 00:01 2024-08-09T14:41:47.397 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:41:47.408 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:41:47.408 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:41:47.505 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:41:47.506 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:41:47.833 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:41:48.009 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-09T14:41:48.067 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-09T14:41:48.594 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-09T14:41:48.680 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-09T14:41:48.744 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-09T14:41:48.772 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 6/6 2024-08-09T14:41:49.552 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 6/6 2024-08-09T14:41:49.552 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 1/6 2024-08-09T14:41:49.553 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-09T14:41:49.553 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-09T14:41:49.553 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-09T14:41:49.553 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-09T14:41:49.819 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-09T14:41:49.819 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:49.819 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:41:49.819 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:41:49.820 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-09T14:41:49.820 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-09T14:41:49.820 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-09T14:41:49.820 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-09T14:41:49.820 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-09T14:41:49.821 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:49.821 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:41:50.089 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-fuse 2024-08-09T14:41:50.637 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:28 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:41:50.763 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:41:50.764 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:50.764 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-08-09T14:41:50.764 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:50.764 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:41:50.764 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 896 k 2024-08-09T14:41:50.764 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-08-09T14:41:50.764 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-09T14:41:50.764 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:50.764 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:41:50.764 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:50.764 INFO:teuthology.orchestra.run.smithi194.stdout:Install 2 Packages 2024-08-09T14:41:50.765 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:50.765 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 976 k 2024-08-09T14:41:50.765 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 2.9 M 2024-08-09T14:41:50.765 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:41:51.180 INFO:teuthology.orchestra.run.smithi194.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 573 kB/s | 80 kB 00:00 2024-08-09T14:41:51.385 INFO:teuthology.orchestra.run.smithi194.stdout:(2/2): ceph-fuse-19.1.0-1345.g0a88632b.el9.x86_ 2.5 MB/s | 896 kB 00:00 2024-08-09T14:41:51.385 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:51.386 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.5 MB/s | 976 kB 00:00 2024-08-09T14:41:51.397 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:41:51.420 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:41:51.420 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:41:51.499 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:41:51.500 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:41:51.684 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:41:51.829 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-09T14:41:51.892 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 2/2 2024-08-09T14:41:52.606 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 2/2 2024-08-09T14:41:52.606 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 1/2 2024-08-09T14:41:52.880 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-09T14:41:52.881 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:52.881 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:41:52.881 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-09T14:41:52.881 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:52.881 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:41:53.133 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-volume 2024-08-09T14:41:53.696 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:31 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:41:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:41:53.829 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:53.830 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-08-09T14:41:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:41:53.830 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 276 k 2024-08-09T14:41:53.830 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:41:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:41:53.830 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 276 k 2024-08-09T14:41:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 1.3 M 2024-08-09T14:41:53.830 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:41:54.074 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-volume-19.1.0-1345.g0a88632b.el9.noarch.rp 1.1 MB/s | 276 kB 00:00 2024-08-09T14:41:54.074 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:54.074 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.1 MB/s | 276 kB 00:00 2024-08-09T14:41:54.075 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:41:54.083 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:41:54.083 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:41:54.108 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:41:54.108 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:41:54.321 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:41:54.384 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-volume-2:19.1.0-1345.g0a88632b.el9.noarch 1/1 2024-08-09T14:41:54.399 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-volume-2:19.1.0-1345.g0a88632b.el9.noarch 1/1 2024-08-09T14:41:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:41:54.399 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-09T14:41:54.400 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:55.414 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-volume-2:19.1.0-1345.g0a88632b.el9.noarch 1/1 2024-08-09T14:41:55.414 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:55.414 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:41:55.414 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:41:55.414 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:55.414 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:41:55.733 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados-devel 2024-08-09T14:41:56.284 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:34 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:41:56.418 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:41:56.418 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:56.418 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:41:56.418 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:56.418 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:41:56.418 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 127 k 2024-08-09T14:41:56.418 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:56.418 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:41:56.418 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:56.419 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:41:56.419 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:56.419 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 127 k 2024-08-09T14:41:56.419 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 456 k 2024-08-09T14:41:56.419 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:41:56.620 INFO:teuthology.orchestra.run.smithi194.stdout:librados-devel-19.1.0-1345.g0a88632b.el9.x86_64 630 kB/s | 127 kB 00:00 2024-08-09T14:41:56.620 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:56.620 INFO:teuthology.orchestra.run.smithi194.stdout:Total 626 kB/s | 127 kB 00:00 2024-08-09T14:41:56.621 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:41:56.626 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:41:56.626 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:41:56.689 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:41:56.689 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:41:56.738 INFO:teuthology.orchestra.run.smithi155.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 883 kB/s | 16 MB 00:18 2024-08-09T14:41:56.739 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:56.739 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.8 MB/s | 38 MB 00:20 2024-08-09T14:41:56.847 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:41:56.968 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:41:57.095 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:41:57.122 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:41:57.122 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:41:57.388 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:41:57.593 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:41:57.593 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:41:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:41:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:57.648 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:41:57.648 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:41:57.649 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:57.649 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:41:57.926 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs2 2024-08-09T14:41:58.346 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:41:58.417 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-08-09T14:41:58.463 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-08-09T14:41:58.481 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:36 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:41:58.560 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-08-09T14:41:58.567 INFO:teuthology.orchestra.run.smithi194.stdout:Package libcephfs2-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:41:58.595 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-08-09T14:41:58.610 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:41:58.611 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-08-09T14:41:58.611 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:41:58.705 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs-devel 2024-08-09T14:41:58.773 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-08-09T14:41:58.802 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-08-09T14:41:58.887 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-08-09T14:41:59.286 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:37 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:41:59.412 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:41:59.413 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:59.413 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:41:59.413 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:59.413 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:41:59.413 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 32 k 2024-08-09T14:41:59.413 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:59.413 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:41:59.413 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:41:59.413 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:41:59.413 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:41:59.413 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 32 k 2024-08-09T14:41:59.414 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 142 k 2024-08-09T14:41:59.414 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:41:59.560 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-08-09T14:41:59.581 INFO:teuthology.orchestra.run.smithi194.stdout:libcephfs-devel-19.1.0-1345.g0a88632b.el9.x86_6 189 kB/s | 32 kB 00:00 2024-08-09T14:41:59.582 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:41:59.582 INFO:teuthology.orchestra.run.smithi194.stdout:Total 187 kB/s | 32 kB 00:00 2024-08-09T14:41:59.582 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:41:59.586 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:41:59.586 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:41:59.604 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:41:59.604 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:41:59.683 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-08-09T14:41:59.687 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:41:59.795 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:00.166 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:00.392 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:00.392 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:00.392 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:42:00.392 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:42:00.392 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:00.392 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:42:00.573 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados2 2024-08-09T14:42:00.913 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-08-09T14:42:01.126 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:39 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:42:01.212 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:42:01.255 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:42:01.256 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-08-09T14:42:01.257 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:42:01.333 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librbd1 2024-08-09T14:42:01.887 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:39 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:42:01.956 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-08-09T14:42:01.972 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:42:01.987 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.0-1345.g0a886 12/12 2024-08-09T14:42:02.015 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:42:02.016 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-08-09T14:42:02.016 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:42:02.090 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rados 2024-08-09T14:42:02.636 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:40 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:42:02.721 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rados-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:42:02.763 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:42:02.766 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-08-09T14:42:02.766 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:42:02.831 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rgw 2024-08-09T14:42:03.383 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:41 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:42:03.469 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rgw-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:42:03.512 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:42:03.513 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-08-09T14:42:03.513 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:42:03.592 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-cephfs 2024-08-09T14:42:04.138 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:42 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:42:04.223 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-cephfs-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:42:04.265 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:42:04.266 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-08-09T14:42:04.266 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:42:04.330 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rbd 2024-08-09T14:42:04.605 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.0-1345.g0a886 12/12 2024-08-09T14:42:04.605 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.0-1345.g0a886 1/12 2024-08-09T14:42:04.605 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-08-09T14:42:04.605 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-08-09T14:42:04.605 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-08-09T14:42:04.605 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-08-09T14:42:04.605 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-08-09T14:42:04.605 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-08-09T14:42:04.605 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-08-09T14:42:04.605 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-08-09T14:42:04.606 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-08-09T14:42:04.606 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-08-09T14:42:04.879 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:42 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:42:04.964 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rbd-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:42:05.007 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:42:05.008 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-08-09T14:42:05.008 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:42:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-08-09T14:42:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:05.028 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:42:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:42:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-08-09T14:42:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-08-09T14:42:05.028 INFO:teuthology.orchestra.run.smithi155.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-08-09T14:42:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-08-09T14:42:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-08-09T14:42:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.21-2.el9.x86_64 2024-08-09T14:42:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-08-09T14:42:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-08-09T14:42:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-08-09T14:42:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-08-09T14:42:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-08-09T14:42:05.029 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:05.029 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:05.072 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-fuse 2024-08-09T14:42:05.422 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-rook 2024-08-09T14:42:05.615 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:43 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:42:05.742 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:42:05.742 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:05.743 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-08-09T14:42:05.743 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:05.743 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:42:05.743 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 85 k 2024-08-09T14:42:05.743 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:05.743 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:42:05.743 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:05.743 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:42:05.743 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:05.743 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 85 k 2024-08-09T14:42:05.743 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 231 k 2024-08-09T14:42:05.744 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:42:05.950 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-fuse-19.1.0-1345.g0a88632b.el9.x86_64.rpm 411 kB/s | 85 kB 00:00 2024-08-09T14:42:05.951 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:05.951 INFO:teuthology.orchestra.run.smithi194.stdout:Total 409 kB/s | 85 kB 00:00 2024-08-09T14:42:05.951 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:42:05.958 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:42:05.958 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:42:05.986 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:39 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:06.022 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:42:06.023 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:42:06.114 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:06.115 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:06.116 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-08-09T14:42:06.116 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:06.116 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:42:06.116 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 49 k 2024-08-09T14:42:06.116 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-08-09T14:42:06.116 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-08-09T14:42:06.116 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-08-09T14:42:06.116 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:2.32.0-1.el9 epel 216 k 2024-08-09T14:42:06.116 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-08-09T14:42:06.116 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-08-09T14:42:06.116 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-08-09T14:42:06.116 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-08-09T14:42:06.117 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-08-09T14:42:06.117 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-08-09T14:42:06.117 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-08-09T14:42:06.117 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-08-09T14:42:06.117 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-08-09T14:42:06.117 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:06.117 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:42:06.117 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:06.117 INFO:teuthology.orchestra.run.smithi155.stdout:Install 13 Packages 2024-08-09T14:42:06.117 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:06.118 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 2.2 M 2024-08-09T14:42:06.118 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 26 M 2024-08-09T14:42:06.118 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:42:06.165 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:42:06.274 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:06.505 INFO:teuthology.orchestra.run.smithi155.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 210 kB/s | 19 kB 00:00 2024-08-09T14:42:06.522 INFO:teuthology.orchestra.run.smithi155.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 236 kB/s | 26 kB 00:00 2024-08-09T14:42:06.622 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:06.622 INFO:teuthology.orchestra.run.smithi155.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-08-09T14:42:06.648 INFO:teuthology.orchestra.run.smithi155.stdout:(4/13): ceph-mgr-rook-19.1.0-1345.g0a88632b.el9 210 kB/s | 49 kB 00:00 2024-08-09T14:42:06.673 INFO:teuthology.orchestra.run.smithi155.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-08-09T14:42:06.698 INFO:teuthology.orchestra.run.smithi155.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-08-09T14:42:06.774 INFO:teuthology.orchestra.run.smithi155.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 427 kB/s | 54 kB 00:00 2024-08-09T14:42:06.799 INFO:teuthology.orchestra.run.smithi155.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 256 kB/s | 32 kB 00:00 2024-08-09T14:42:06.849 INFO:teuthology.orchestra.run.smithi155.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 112 kB/s | 14 kB 00:00 2024-08-09T14:42:06.874 INFO:teuthology.orchestra.run.smithi155.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-08-09T14:42:06.899 INFO:teuthology.orchestra.run.smithi155.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-08-09T14:42:06.903 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:06.903 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:06.903 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:42:06.903 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:42:06.903 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:06.904 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:42:06.925 INFO:teuthology.orchestra.run.smithi155.stdout:(12/13): python3-google-auth-2.32.0-1.el9.noarc 1.4 MB/s | 216 kB 00:00 2024-08-09T14:42:06.958 INFO:teuthology.orchestra.run.smithi155.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.6 MB/s | 1.0 MB 00:00 2024-08-09T14:42:06.961 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:06.961 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.6 MB/s | 2.2 MB 00:00 2024-08-09T14:42:07.023 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:42:07.038 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:42:07.038 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:42:07.124 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:42:07.126 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:42:07.141 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-mirror 2024-08-09T14:42:07.638 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:42:07.716 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:45 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:42:07.817 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-08-09T14:42:07.850 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:42:07.850 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:07.851 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:42:07.851 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:07.851 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:42:07.851 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 3.1 M 2024-08-09T14:42:07.851 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:07.851 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:42:07.851 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:07.851 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:42:07.851 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:07.852 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 3.1 M 2024-08-09T14:42:07.852 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 13 M 2024-08-09T14:42:07.852 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:42:07.877 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-08-09T14:42:07.932 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-08-09T14:42:07.969 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-08-09T14:42:08.011 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-08-09T14:42:08.119 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-08-09T14:42:08.224 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:2.32.0-1.el9.noarch 7/13 2024-08-09T14:42:08.229 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-mirror-19.1.0-1345.g0a88632b.el9.x86_64.rpm 8.2 MB/s | 3.1 MB 00:00 2024-08-09T14:42:08.230 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:08.230 INFO:teuthology.orchestra.run.smithi194.stdout:Total 8.2 MB/s | 3.1 MB 00:00 2024-08-09T14:42:08.230 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:42:08.239 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:42:08.240 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:42:08.280 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-08-09T14:42:08.321 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:42:08.322 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:42:08.758 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:42:08.849 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:08.874 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:08.875 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:42:08.875 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-09T14:42:08.875 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-09T14:42:08.875 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-09T14:42:08.875 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:09.012 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-08-09T14:42:09.077 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-08-09T14:42:09.111 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-08-09T14:42:09.159 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-08-09T14:42:09.181 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:19.1.0-1345.g0a88632b.el9.noarch 13/13 2024-08-09T14:42:09.919 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.0-1345.g0a88632b.el9.noarch 13/13 2024-08-09T14:42:09.919 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:19.1.0-1345.g0a88632b.el9.noarch 1/13 2024-08-09T14:42:09.919 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-08-09T14:42:09.919 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-08-09T14:42:09.920 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-08-09T14:42:09.920 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-08-09T14:42:09.920 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-08-09T14:42:09.920 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-08-09T14:42:09.920 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-08-09T14:42:09.920 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-08-09T14:42:09.920 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:2.32.0-1.el9.noarch 10/13 2024-08-09T14:42:09.920 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-08-09T14:42:09.920 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-08-09T14:42:10.018 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:10.018 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:10.018 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:42:10.018 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:42:10.018 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:10.019 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:42:10.234 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-nbd 2024-08-09T14:42:10.259 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-08-09T14:42:10.259 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:10.259 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:42:10.259 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:42:10.259 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-08-09T14:42:10.259 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-08-09T14:42:10.259 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:2.32.0-1.el9.noarch 2024-08-09T14:42:10.259 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-08-09T14:42:10.260 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-08-09T14:42:10.260 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-08-09T14:42:10.260 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-08-09T14:42:10.260 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-08-09T14:42:10.260 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-08-09T14:42:10.260 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-08-09T14:42:10.260 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-4.9-2.el9.noarch 2024-08-09T14:42:10.260 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-08-09T14:42:10.260 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:10.260 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:10.497 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-cephadm 2024-08-09T14:42:10.809 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:48 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:42:10.938 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:42:10.939 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:10.939 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-08-09T14:42:10.939 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:10.939 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:42:10.940 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 170 k 2024-08-09T14:42:10.940 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:10.940 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:42:10.940 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:10.940 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:42:10.940 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:10.940 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 170 k 2024-08-09T14:42:10.940 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 494 k 2024-08-09T14:42:10.940 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:42:11.040 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:45 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:11.169 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:11.169 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-08-09T14:42:11.170 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:11.170 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:42:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 146 k 2024-08-09T14:42:11.170 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-08-09T14:42:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-08-09T14:42:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-08-09T14:42:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-08-09T14:42:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-08-09T14:42:11.170 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-08-09T14:42:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:11.171 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:42:11.171 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:11.171 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2024-08-09T14:42:11.171 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:11.172 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 7.0 M 2024-08-09T14:42:11.172 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 33 M 2024-08-09T14:42:11.172 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:42:11.172 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-nbd-19.1.0-1345.g0a88632b.el9.x86_64.rpm 739 kB/s | 170 kB 00:00 2024-08-09T14:42:11.172 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:11.173 INFO:teuthology.orchestra.run.smithi194.stdout:Total 735 kB/s | 170 kB 00:00 2024-08-09T14:42:11.173 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:42:11.179 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:42:11.179 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:42:11.243 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:42:11.243 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:42:11.399 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:42:11.509 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:11.534 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-08-09T14:42:11.601 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.1 MB/s | 548 kB 00:00 2024-08-09T14:42:11.627 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): ceph-mgr-cephadm-19.1.0-1345.g0a88632b.e 651 kB/s | 146 kB 00:00 2024-08-09T14:42:11.652 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-08-09T14:42:11.711 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 891 kB/s | 74 kB 00:00 2024-08-09T14:42:11.923 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-nbd-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:12.177 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:12.178 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:12.178 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:42:12.178 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:42:12.178 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:12.179 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:42:12.429 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-base-debuginfo 2024-08-09T14:42:12.903 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.0 MB/s | 6.0 MB 00:01 2024-08-09T14:42:12.906 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:12.906 INFO:teuthology.orchestra.run.smithi155.stdout:Total 4.0 MB/s | 7.0 MB 00:01 2024-08-09T14:42:12.978 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:42:12.988 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:50 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:42:12.989 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:42:12.989 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:42:13.087 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:42:13.087 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:42:13.121 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:42:13.121 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:13.121 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:42:13.122 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:13.122 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:42:13.122 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 69 M 2024-08-09T14:42:13.122 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-08-09T14:42:13.122 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 23 M 2024-08-09T14:42:13.122 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:13.122 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:42:13.122 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:13.122 INFO:teuthology.orchestra.run.smithi194.stdout:Install 2 Packages 2024-08-09T14:42:13.122 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:13.123 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 92 M 2024-08-09T14:42:13.123 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 505 M 2024-08-09T14:42:13.123 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:42:13.388 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:42:13.550 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-08-09T14:42:13.605 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-08-09T14:42:14.148 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-08-09T14:42:14.250 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-08-09T14:42:14.314 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-08-09T14:42:14.315 INFO:teuthology.orchestra.run.smithi194.stdout:(1/2): ceph-debuginfo-19.1.0-1345.g0a88632b.el9 20 MB/s | 23 MB 00:01 2024-08-09T14:42:14.344 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 6/6 2024-08-09T14:42:15.075 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 6/6 2024-08-09T14:42:15.076 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 1/6 2024-08-09T14:42:15.076 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-08-09T14:42:15.076 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-08-09T14:42:15.076 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-08-09T14:42:15.076 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-08-09T14:42:15.341 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-08-09T14:42:15.341 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:15.341 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:42:15.342 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:42:15.342 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-08-09T14:42:15.342 INFO:teuthology.orchestra.run.smithi155.stdout: python3-babel-2.9.1-2.el9.noarch 2024-08-09T14:42:15.342 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-08-09T14:42:15.342 INFO:teuthology.orchestra.run.smithi155.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-08-09T14:42:15.342 INFO:teuthology.orchestra.run.smithi155.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-08-09T14:42:15.342 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:15.342 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:15.619 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-fuse 2024-08-09T14:42:15.657 INFO:teuthology.orchestra.run.smithi194.stdout:(2/2): ceph-base-debuginfo-19.1.0-1345.g0a88632 27 MB/s | 69 MB 00:02 2024-08-09T14:42:15.659 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:15.660 INFO:teuthology.orchestra.run.smithi194.stdout:Total 36 MB/s | 92 MB 00:02 2024-08-09T14:42:15.660 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:42:15.664 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:42:15.665 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:42:16.173 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:50 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:16.255 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:42:16.256 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:42:16.301 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:16.302 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:16.302 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-08-09T14:42:16.302 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:16.302 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:42:16.302 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 896 k 2024-08-09T14:42:16.302 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-08-09T14:42:16.303 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-09T14:42:16.303 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:16.303 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:42:16.303 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:16.303 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2024-08-09T14:42:16.303 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:16.303 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 976 k 2024-08-09T14:42:16.304 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 2.9 M 2024-08-09T14:42:16.304 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:42:16.529 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 423 kB/s | 80 kB 00:00 2024-08-09T14:42:16.629 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): ceph-fuse-19.1.0-1345.g0a88632b.el9.x86_ 3.0 MB/s | 896 kB 00:00 2024-08-09T14:42:16.629 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:16.630 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.9 MB/s | 976 kB 00:00 2024-08-09T14:42:16.641 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:42:16.663 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:42:16.663 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:42:16.741 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:42:16.742 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:42:16.933 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:42:17.074 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-09T14:42:17.136 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 2/2 2024-08-09T14:42:17.818 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 2/2 2024-08-09T14:42:17.818 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 1/2 2024-08-09T14:42:18.099 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-09T14:42:18.100 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:18.100 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:42:18.100 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-09T14:42:18.100 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:18.100 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:18.362 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-volume 2024-08-09T14:42:18.925 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:52 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:19.059 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:19.060 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:19.060 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2024-08-09T14:42:19.060 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:19.060 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:42:19.060 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume noarch 2:19.1.0-1345.g0a88632b.el9 ceph-noarch 276 k 2024-08-09T14:42:19.060 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:19.060 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:42:19.061 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:19.061 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:42:19.061 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:19.061 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 276 k 2024-08-09T14:42:19.061 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 1.3 M 2024-08-09T14:42:19.061 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:42:19.303 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-volume-19.1.0-1345.g0a88632b.el9.noarch.rp 1.1 MB/s | 276 kB 00:00 2024-08-09T14:42:19.304 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:19.304 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.1 MB/s | 276 kB 00:00 2024-08-09T14:42:19.304 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:42:19.312 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:42:19.312 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:42:19.339 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:42:19.339 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:42:19.456 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:42:19.555 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:42:19.630 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-volume-2:19.1.0-1345.g0a88632b.el9.noarch 1/1 2024-08-09T14:42:19.646 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-volume-2:19.1.0-1345.g0a88632b.el9.noarch 1/1 2024-08-09T14:42:19.646 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:42:19.646 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-08-09T14:42:19.646 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:20.682 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-volume-2:19.1.0-1345.g0a88632b.el9.noarch 1/1 2024-08-09T14:42:20.683 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:20.683 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:42:20.683 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-volume-2:19.1.0-1345.g0a88632b.el9.noarch 2024-08-09T14:42:20.683 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:20.683 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:20.947 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados-devel 2024-08-09T14:42:21.512 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:55 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:21.643 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:21.644 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:21.644 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:42:21.644 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:21.644 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:42:21.644 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 127 k 2024-08-09T14:42:21.644 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:21.644 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:42:21.645 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:21.645 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:42:21.645 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:21.645 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 127 k 2024-08-09T14:42:21.645 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 456 k 2024-08-09T14:42:21.645 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:42:21.877 INFO:teuthology.orchestra.run.smithi155.stdout:librados-devel-19.1.0-1345.g0a88632b.el9.x86_64 547 kB/s | 127 kB 00:00 2024-08-09T14:42:21.877 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:21.878 INFO:teuthology.orchestra.run.smithi155.stdout:Total 544 kB/s | 127 kB 00:00 2024-08-09T14:42:21.878 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:42:21.883 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:42:21.883 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:42:21.946 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:42:21.947 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:42:22.100 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:42:22.199 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:22.620 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:22.874 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:22.874 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:22.874 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:42:22.874 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:42:22.874 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:22.875 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:23.130 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs2 2024-08-09T14:42:23.688 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:57 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:23.776 INFO:teuthology.orchestra.run.smithi155.stdout:Package libcephfs2-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:42:23.819 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:23.820 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-08-09T14:42:23.820 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:23.888 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs-devel 2024-08-09T14:42:24.433 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:58 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:24.564 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:24.565 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:24.565 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:42:24.566 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:24.566 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:42:24.566 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 32 k 2024-08-09T14:42:24.566 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:24.566 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:42:24.567 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:24.567 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:42:24.567 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:24.567 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 32 k 2024-08-09T14:42:24.567 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 142 k 2024-08-09T14:42:24.567 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:42:24.732 INFO:teuthology.orchestra.run.smithi155.stdout:libcephfs-devel-19.1.0-1345.g0a88632b.el9.x86_6 191 kB/s | 32 kB 00:00 2024-08-09T14:42:24.733 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:24.733 INFO:teuthology.orchestra.run.smithi155.stdout:Total 189 kB/s | 32 kB 00:00 2024-08-09T14:42:24.733 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:42:24.736 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:42:24.736 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:42:24.757 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:42:24.758 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:42:24.839 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:42:24.946 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:25.347 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:25.562 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:25.562 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:25.562 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:42:25.563 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:42:25.563 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:25.563 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:25.728 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados2 2024-08-09T14:42:26.284 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:00 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:26.370 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:42:26.412 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:26.413 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-08-09T14:42:26.413 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:26.477 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librbd1 2024-08-09T14:42:26.642 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 1/2 2024-08-09T14:42:26.716 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 2/2 2024-08-09T14:42:27.032 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:01 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:27.117 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:42:27.160 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:27.161 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-08-09T14:42:27.161 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:27.226 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rados 2024-08-09T14:42:27.771 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:01 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:27.857 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rados-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:42:27.899 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:27.900 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-08-09T14:42:27.900 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:27.963 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rgw 2024-08-09T14:42:28.511 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:02 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:28.597 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rgw-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:42:28.639 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:28.640 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-08-09T14:42:28.640 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:28.711 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-cephfs 2024-08-09T14:42:29.256 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:03 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:29.341 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-cephfs-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:42:29.384 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:29.385 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-08-09T14:42:29.385 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:29.449 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rbd 2024-08-09T14:42:29.997 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 2/2 2024-08-09T14:42:29.997 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/2 2024-08-09T14:42:30.000 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:03 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:30.086 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rbd-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:42:30.128 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:30.129 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-08-09T14:42:30.129 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:30.191 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-fuse 2024-08-09T14:42:30.270 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2/2 2024-08-09T14:42:30.270 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:30.270 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:42:30.270 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:42:30.270 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:42:30.270 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:30.270 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:42:30.497 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-common-debuginfo 2024-08-09T14:42:30.739 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:04 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:30.865 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:30.866 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:30.866 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-08-09T14:42:30.866 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:30.866 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:42:30.866 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 85 k 2024-08-09T14:42:30.866 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:30.866 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:42:30.867 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:30.867 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:42:30.867 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:30.867 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 85 k 2024-08-09T14:42:30.867 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 231 k 2024-08-09T14:42:30.867 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:42:31.045 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:09 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:42:31.085 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-fuse-19.1.0-1345.g0a88632b.el9.x86_64.rpm 390 kB/s | 85 kB 00:00 2024-08-09T14:42:31.085 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:31.085 INFO:teuthology.orchestra.run.smithi155.stdout:Total 388 kB/s | 85 kB 00:00 2024-08-09T14:42:31.086 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:42:31.093 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:42:31.093 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:42:31.155 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:42:31.155 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:42:31.173 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:42:31.174 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:31.174 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:42:31.174 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:31.174 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:42:31.174 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 454 M 2024-08-09T14:42:31.174 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:31.174 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:42:31.174 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:42:31.174 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:42:31.174 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:42:31.175 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 454 M 2024-08-09T14:42:31.175 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 1.8 G 2024-08-09T14:42:31.175 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:42:31.310 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:42:31.407 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:31.770 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:32.019 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:32.019 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:32.019 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:42:32.020 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:42:32.020 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:32.020 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:32.269 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-mirror 2024-08-09T14:42:32.841 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:06 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:32.979 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:32.980 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:32.980 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:42:32.981 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:32.981 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:42:32.981 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 3.1 M 2024-08-09T14:42:32.981 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:32.981 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:42:32.981 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:32.981 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:42:32.981 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:32.981 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 3.1 M 2024-08-09T14:42:32.981 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 13 M 2024-08-09T14:42:32.981 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:42:33.784 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-mirror-19.1.0-1345.g0a88632b.el9.x86_64.rpm 3.9 MB/s | 3.1 MB 00:00 2024-08-09T14:42:33.784 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:33.785 INFO:teuthology.orchestra.run.smithi155.stdout:Total 3.9 MB/s | 3.1 MB 00:00 2024-08-09T14:42:33.785 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:42:33.794 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:42:33.794 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:42:33.879 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:42:33.879 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:42:34.311 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:42:34.393 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:34.420 INFO:teuthology.orchestra.run.smithi155.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-09T14:42:34.420 INFO:teuthology.orchestra.run.smithi155.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-09T14:42:34.420 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-09T14:42:34.420 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-09T14:42:34.420 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:35.538 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:35.538 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:35.538 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:42:35.538 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:42:35.538 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:35.538 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:35.788 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-nbd 2024-08-09T14:42:36.358 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:10 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:36.486 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:36.487 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:36.487 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2024-08-09T14:42:36.487 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:36.487 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:42:36.487 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 170 k 2024-08-09T14:42:36.487 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:36.487 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:42:36.488 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:36.488 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:42:36.488 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:36.488 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 170 k 2024-08-09T14:42:36.488 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 494 k 2024-08-09T14:42:36.488 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:42:36.884 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-nbd-19.1.0-1345.g0a88632b.el9.x86_64.rpm 430 kB/s | 170 kB 00:00 2024-08-09T14:42:36.885 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:36.885 INFO:teuthology.orchestra.run.smithi155.stdout:Total 428 kB/s | 170 kB 00:00 2024-08-09T14:42:36.885 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:42:36.893 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:42:36.893 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:42:36.959 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:42:36.959 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:42:37.109 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:42:37.215 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:37.618 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-nbd-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:37.854 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:42:37.854 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:37.854 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:42:37.854 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:42:37.854 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:37.854 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:38.084 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base-debuginfo 2024-08-09T14:42:38.641 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:12 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:38.773 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:38.774 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:38.774 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:42:38.774 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:38.774 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:42:38.774 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 69 M 2024-08-09T14:42:38.774 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2024-08-09T14:42:38.774 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 23 M 2024-08-09T14:42:38.775 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:38.775 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:42:38.775 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:38.775 INFO:teuthology.orchestra.run.smithi155.stdout:Install 2 Packages 2024-08-09T14:42:38.775 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:38.775 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 92 M 2024-08-09T14:42:38.775 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 505 M 2024-08-09T14:42:38.776 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:42:40.897 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): ceph-debuginfo-19.1.0-1345.g0a88632b.el9 11 MB/s | 23 MB 00:02 2024-08-09T14:42:42.230 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): ceph-base-debuginfo-19.1.0-1345.g0a88632 20 MB/s | 69 MB 00:03 2024-08-09T14:42:42.232 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:42.232 INFO:teuthology.orchestra.run.smithi155.stdout:Total 27 MB/s | 92 MB 00:03 2024-08-09T14:42:42.233 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:42:42.237 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:42:42.238 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:42:42.826 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:42:42.827 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:42:43.020 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-common-debuginfo-19.1.0-1345.g0a88632b.el9 38 MB/s | 454 MB 00:11 2024-08-09T14:42:43.020 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:42:43.021 INFO:teuthology.orchestra.run.smithi194.stdout:Total 38 MB/s | 454 MB 00:11 2024-08-09T14:42:43.021 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:42:43.025 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:42:43.025 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:42:45.716 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:42:45.717 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:42:46.033 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:42:53.234 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 1/2 2024-08-09T14:42:53.306 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 2/2 2024-08-09T14:42:56.864 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 2/2 2024-08-09T14:42:56.864 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/2 2024-08-09T14:42:57.082 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2/2 2024-08-09T14:42:57.082 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:57.083 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:42:57.083 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:42:57.083 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:42:57.083 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:57.083 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:42:57.305 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-common-debuginfo 2024-08-09T14:42:57.858 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:31 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:42:57.987 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:42:57.987 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:57.987 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:42:57.988 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:57.988 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:42:57.988 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 454 M 2024-08-09T14:42:57.988 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:57.988 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:42:57.988 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:42:57.988 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:42:57.988 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:42:57.988 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 454 M 2024-08-09T14:42:57.988 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 1.8 G 2024-08-09T14:42:57.988 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:43:09.836 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-common-debuginfo-19.1.0-1345.g0a88632b.el9 38 MB/s | 454 MB 00:11 2024-08-09T14:43:09.836 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:43:09.837 INFO:teuthology.orchestra.run.smithi155.stdout:Total 38 MB/s | 454 MB 00:11 2024-08-09T14:43:09.837 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:43:09.841 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:43:09.842 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:43:12.525 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:43:12.525 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:43:31.506 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:43:31.613 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:43:38.712 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:43:38.945 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:43:38.945 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:43:38.945 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:43:38.945 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:43:38.945 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:43:38.945 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:43:39.229 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-08-09T14:43:39.779 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:03:17 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:43:39.907 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:43:39.907 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-08-09T14:43:39.908 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:43:39.908 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-08-09T14:43:39.908 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:43:39.908 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 1.5 M 2024-08-09T14:43:39.908 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:43:39.908 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:43:39.908 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================== 2024-08-09T14:43:39.908 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:43:39.908 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:43:39.908 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 1.5 M 2024-08-09T14:43:39.908 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 6.6 M 2024-08-09T14:43:39.908 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:43:40.225 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-immutable-object-cache-debuginfo-19.1.0-13 4.6 MB/s | 1.5 MB 00:00 2024-08-09T14:43:40.226 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:43:40.226 INFO:teuthology.orchestra.run.smithi194.stdout:Total 4.6 MB/s | 1.5 MB 00:00 2024-08-09T14:43:40.226 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:43:40.229 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:43:40.230 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:43:40.274 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:43:40.275 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:43:40.554 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:43:40.650 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.1.0-1345. 1/1 2024-08-09T14:43:41.039 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.1.0-1345. 1/1 2024-08-09T14:43:41.276 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.1.0-1345. 1/1 2024-08-09T14:43:41.276 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:43:41.277 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:43:41.277 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:43:41.277 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:43:41.277 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:43:41.451 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw-debuginfo 2024-08-09T14:43:42.008 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:03:20 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:43:42.137 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:43:42.138 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:43:42.138 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:43:42.138 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:43:42.138 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:43:42.139 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 189 M 2024-08-09T14:43:42.139 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:43:42.139 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:43:42.139 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:43:42.139 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:43:42.139 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:43:42.139 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 189 M 2024-08-09T14:43:42.139 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 803 M 2024-08-09T14:43:42.139 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:43:52.944 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-radosgw-debuginfo-19.1.0-1345.g0a88632b.el 17 MB/s | 189 MB 00:10 2024-08-09T14:43:52.945 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:43:52.946 INFO:teuthology.orchestra.run.smithi194.stdout:Total 17 MB/s | 189 MB 00:10 2024-08-09T14:43:52.946 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:43:52.950 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:43:52.950 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:43:54.089 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:43:54.089 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:43:58.362 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:43:58.472 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:44:07.049 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:44:07.290 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:44:07.290 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:44:07.290 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:44:07.290 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:44:07.290 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:44:07.290 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:44:07.523 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-08-09T14:44:08.074 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:42 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:44:08.204 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:44:08.205 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-08-09T14:44:08.205 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:44:08.205 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-08-09T14:44:08.205 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:44:08.205 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 1.5 M 2024-08-09T14:44:08.205 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:44:08.205 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:44:08.205 INFO:teuthology.orchestra.run.smithi155.stdout:======================================================================================== 2024-08-09T14:44:08.205 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:44:08.205 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:44:08.206 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 1.5 M 2024-08-09T14:44:08.206 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 6.6 M 2024-08-09T14:44:08.206 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:44:08.493 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-immutable-object-cache-debuginfo-19.1.0-13 5.1 MB/s | 1.5 MB 00:00 2024-08-09T14:44:08.494 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:44:08.494 INFO:teuthology.orchestra.run.smithi155.stdout:Total 5.1 MB/s | 1.5 MB 00:00 2024-08-09T14:44:08.494 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:44:08.498 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:44:08.498 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:44:08.541 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:44:08.542 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:44:08.802 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:44:08.899 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.1.0-1345. 1/1 2024-08-09T14:44:09.300 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.1.0-1345. 1/1 2024-08-09T14:44:09.515 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.1.0-1345. 1/1 2024-08-09T14:44:09.515 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:44:09.515 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:44:09.515 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:44:09.515 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:44:09.516 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:44:09.684 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw-debuginfo 2024-08-09T14:44:10.254 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:03:44 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:44:10.382 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:44:10.383 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:44:10.383 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:44:10.383 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:44:10.383 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:44:10.384 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 189 M 2024-08-09T14:44:10.384 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:44:10.384 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:44:10.384 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:44:10.384 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:44:10.384 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:44:10.384 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 189 M 2024-08-09T14:44:10.384 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 803 M 2024-08-09T14:44:10.384 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:44:13.382 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:44:13.490 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x 1/1 2024-08-09T14:44:15.438 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-radosgw-debuginfo-19.1.0-1345.g0a88632b.el 37 MB/s | 189 MB 00:05 2024-08-09T14:44:15.439 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:44:15.439 INFO:teuthology.orchestra.run.smithi155.stdout:Total 37 MB/s | 189 MB 00:05 2024-08-09T14:44:15.440 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:44:15.444 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:44:15.444 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:44:16.581 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:44:16.582 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:44:18.139 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x 1/1 2024-08-09T14:44:18.360 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x 1/1 2024-08-09T14:44:18.360 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:44:18.360 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:44:18.360 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:44:18.360 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:44:18.361 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:44:18.596 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-test-debuginfo 2024-08-09T14:44:19.144 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:03:57 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:44:19.272 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:44:19.273 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:44:19.273 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:44:19.274 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:44:19.274 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:44:19.274 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 697 M 2024-08-09T14:44:19.274 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:44:19.274 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:44:19.274 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:44:19.274 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:44:19.274 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:44:19.274 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 697 M 2024-08-09T14:44:19.274 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 3.2 G 2024-08-09T14:44:19.274 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:44:35.925 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:44:36.034 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x 1/1 2024-08-09T14:44:37.134 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-test-debuginfo-19.1.0-1345.g0a88632b.el9.x 39 MB/s | 697 MB 00:17 2024-08-09T14:44:37.134 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:44:37.135 INFO:teuthology.orchestra.run.smithi194.stdout:Total 39 MB/s | 697 MB 00:17 2024-08-09T14:44:37.135 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:44:37.141 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:44:37.141 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:44:40.990 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x 1/1 2024-08-09T14:44:41.210 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x 1/1 2024-08-09T14:44:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:44:41.211 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:44:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:44:41.211 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:44:41.211 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:44:41.244 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:44:41.245 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:44:41.420 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-test-debuginfo 2024-08-09T14:44:41.970 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:04:15 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:44:42.100 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:44:42.101 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:44:42.101 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:44:42.101 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:44:42.101 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:44:42.101 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 697 M 2024-08-09T14:44:42.101 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:44:42.101 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:44:42.101 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:44:42.101 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:44:42.101 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:44:42.101 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 697 M 2024-08-09T14:44:42.102 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 3.2 G 2024-08-09T14:44:42.102 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:45:03.571 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-test-debuginfo-19.1.0-1345.g0a88632b.el9.x 32 MB/s | 697 MB 00:21 2024-08-09T14:45:03.571 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:45:03.572 INFO:teuthology.orchestra.run.smithi155.stdout:Total 32 MB/s | 697 MB 00:21 2024-08-09T14:45:03.572 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:45:03.578 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:45:03.578 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:45:07.680 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:45:07.681 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:45:54.448 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:45:54.610 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:46:03.596 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-test-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:46:03.839 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:46:03.839 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:03.839 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:46:03.839 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:46:03.839 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:03.839 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:46:04.142 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-base-debuginfo 2024-08-09T14:46:04.690 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:05:42 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:46:04.777 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-base-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:46:04.820 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:46:04.820 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-08-09T14:46:04.821 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:46:04.943 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-debuginfo 2024-08-09T14:46:05.492 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:05:43 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:46:05.621 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:46:05.622 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:46:05.622 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:46:05.622 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:46:05.622 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:46:05.622 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 20 M 2024-08-09T14:46:05.622 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:05.622 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:46:05.623 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:46:05.623 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:46:05.623 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:05.623 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 20 M 2024-08-09T14:46:05.623 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 105 M 2024-08-09T14:46:05.623 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:46:06.446 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-mgr-debuginfo-19.1.0-1345.g0a88632b.el9.x8 25 MB/s | 20 MB 00:00 2024-08-09T14:46:06.447 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:46:06.447 INFO:teuthology.orchestra.run.smithi194.stdout:Total 25 MB/s | 20 MB 00:00 2024-08-09T14:46:06.448 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:46:06.452 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:46:06.452 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:46:06.607 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:46:06.607 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:46:08.927 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:46:09.035 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:09.925 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:10.159 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:10.160 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:10.160 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:46:10.160 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:46:10.160 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:10.160 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:46:10.361 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mds-debuginfo 2024-08-09T14:46:10.907 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:05:48 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:46:11.035 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:46:11.037 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:46:11.037 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:46:11.037 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:46:11.037 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:46:11.038 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 54 M 2024-08-09T14:46:11.038 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:11.038 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:46:11.038 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:46:11.038 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:46:11.039 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:11.039 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 54 M 2024-08-09T14:46:11.039 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 196 M 2024-08-09T14:46:11.039 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:46:13.023 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-mds-debuginfo-19.1.0-1345.g0a88632b.el9.x8 27 MB/s | 54 MB 00:01 2024-08-09T14:46:13.024 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:46:13.025 INFO:teuthology.orchestra.run.smithi194.stdout:Total 27 MB/s | 54 MB 00:01 2024-08-09T14:46:13.025 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:46:13.029 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:46:13.029 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:46:13.380 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:46:13.380 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:46:18.712 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:46:18.820 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:20.275 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:20.519 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:20.519 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:20.519 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:46:20.520 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:46:20.520 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:20.520 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:46:20.723 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mon-debuginfo 2024-08-09T14:46:20.963 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:46:21.269 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:05:59 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:46:21.283 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:46:21.398 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:46:21.399 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:46:21.399 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:46:21.399 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:46:21.399 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:46:21.399 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 99 M 2024-08-09T14:46:21.399 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:21.399 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:46:21.400 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:46:21.400 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:46:21.400 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:21.400 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 99 M 2024-08-09T14:46:21.400 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 451 M 2024-08-09T14:46:21.400 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:46:24.158 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-mon-debuginfo-19.1.0-1345.g0a88632b.el9.x8 36 MB/s | 99 MB 00:02 2024-08-09T14:46:24.159 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:46:24.159 INFO:teuthology.orchestra.run.smithi194.stdout:Total 36 MB/s | 99 MB 00:02 2024-08-09T14:46:24.160 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:46:24.164 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:46:24.164 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:46:24.777 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:46:24.778 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:46:29.202 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-test-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:46:29.430 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:46:29.430 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:46:29.430 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:46:29.430 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:46:29.430 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:46:29.430 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:46:29.724 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base-debuginfo 2024-08-09T14:46:30.271 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:04 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:46:30.357 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-base-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 is already installed. 2024-08-09T14:46:30.400 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:46:30.401 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2024-08-09T14:46:30.401 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:46:30.462 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-debuginfo 2024-08-09T14:46:31.008 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:05 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:46:31.136 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:46:31.136 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:46:31.137 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:46:31.137 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:46:31.137 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:46:31.137 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 20 M 2024-08-09T14:46:31.137 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:46:31.137 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:46:31.137 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:46:31.137 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:46:31.137 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:46:31.137 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 20 M 2024-08-09T14:46:31.137 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 105 M 2024-08-09T14:46:31.138 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:46:32.345 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-mgr-debuginfo-19.1.0-1345.g0a88632b.el9.x8 17 MB/s | 20 MB 00:01 2024-08-09T14:46:32.346 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:46:32.347 INFO:teuthology.orchestra.run.smithi155.stdout:Total 17 MB/s | 20 MB 00:01 2024-08-09T14:46:32.347 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:46:32.351 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:46:32.351 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:46:32.508 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:46:32.508 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:46:34.826 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:46:34.934 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:35.093 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:46:35.195 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:35.906 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:36.125 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:36.125 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:46:36.125 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:46:36.125 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:46:36.125 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:46:36.125 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:46:36.292 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mds-debuginfo 2024-08-09T14:46:36.840 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:10 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:46:36.970 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:46:36.971 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:46:36.971 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 54 M 2024-08-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 54 M 2024-08-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 196 M 2024-08-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:46:38.010 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:38.269 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:46:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:46:38.269 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:38.269 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:46:38.486 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-osd-debuginfo 2024-08-09T14:46:38.690 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-mds-debuginfo-19.1.0-1345.g0a88632b.el9.x8 31 MB/s | 54 MB 00:01 2024-08-09T14:46:38.691 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:46:38.691 INFO:teuthology.orchestra.run.smithi155.stdout:Total 31 MB/s | 54 MB 00:01 2024-08-09T14:46:38.691 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:46:38.695 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:46:38.695 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:46:39.036 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:06:17 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:46:39.043 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:46:39.044 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:46:39.167 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:46:39.168 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:46:39.168 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:46:39.168 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:46:39.168 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:46:39.169 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 290 M 2024-08-09T14:46:39.169 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:39.169 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:46:39.169 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:46:39.169 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:46:39.169 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:46:39.169 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 290 M 2024-08-09T14:46:39.169 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 1.2 G 2024-08-09T14:46:39.169 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:46:44.372 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:46:44.461 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:45.936 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:46.156 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:46:46.156 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:46:46.156 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:46:46.156 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:46:46.156 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:46:46.156 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:46:46.326 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mon-debuginfo 2024-08-09T14:46:46.877 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:20 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:46:46.992 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-osd-debuginfo-19.1.0-1345.g0a88632b.el9.x8 37 MB/s | 290 MB 00:07 2024-08-09T14:46:46.993 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:46:46.993 INFO:teuthology.orchestra.run.smithi194.stdout:Total 37 MB/s | 290 MB 00:07 2024-08-09T14:46:46.994 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:46:46.998 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:46:46.998 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:46:47.006 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:46:47.007 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:46:47.007 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:46:47.007 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:46:47.007 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:46:47.007 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 99 M 2024-08-09T14:46:47.007 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:46:47.008 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:46:47.008 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:46:47.008 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:46:47.008 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:46:47.008 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 99 M 2024-08-09T14:46:47.008 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 451 M 2024-08-09T14:46:47.008 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:46:48.720 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:46:48.720 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:46:49.917 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-mon-debuginfo-19.1.0-1345.g0a88632b.el9.x8 34 MB/s | 99 MB 00:02 2024-08-09T14:46:49.918 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:46:49.918 INFO:teuthology.orchestra.run.smithi155.stdout:Total 34 MB/s | 99 MB 00:02 2024-08-09T14:46:49.918 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:46:49.922 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:46:49.923 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:46:50.536 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:46:50.536 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:47:00.864 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:47:00.964 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:47:03.891 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:47:04.138 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:47:04.138 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:04.138 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:47:04.138 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:47:04.138 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:04.138 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:47:04.315 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-osd-debuginfo 2024-08-09T14:47:04.862 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:06:38 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:47:04.991 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:47:04.992 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:47:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:47:04.992 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:47:04.992 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:47:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 290 M 2024-08-09T14:47:04.992 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:04.992 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:47:04.992 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:47:04.992 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:47:04.993 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:04.993 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 290 M 2024-08-09T14:47:04.993 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 1.2 G 2024-08-09T14:47:04.993 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:47:14.410 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-osd-debuginfo-19.1.0-1345.g0a88632b.el9.x8 31 MB/s | 290 MB 00:09 2024-08-09T14:47:14.411 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:47:14.411 INFO:teuthology.orchestra.run.smithi155.stdout:Total 31 MB/s | 290 MB 00:09 2024-08-09T14:47:14.412 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:47:14.416 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:47:14.416 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:47:16.137 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:47:16.137 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:47:18.335 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:47:18.419 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:47:25.628 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:47:25.882 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:47:25.883 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:25.883 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:47:25.883 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:47:25.883 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:25.883 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:47:26.109 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-fuse-debuginfo 2024-08-09T14:47:26.660 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:07:04 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:47:26.789 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:47:26.790 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:26.790 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:47:26.790 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:26.790 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:47:26.790 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 6.1 M 2024-08-09T14:47:26.790 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:26.790 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:47:26.791 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:26.791 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:47:26.791 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:26.791 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 6.1 M 2024-08-09T14:47:26.791 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 29 M 2024-08-09T14:47:26.791 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:47:27.216 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-fuse-debuginfo-19.1.0-1345.g0a88632b.el9.x 14 MB/s | 6.1 MB 00:00 2024-08-09T14:47:27.216 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:47:27.217 INFO:teuthology.orchestra.run.smithi194.stdout:Total 14 MB/s | 6.1 MB 00:00 2024-08-09T14:47:27.217 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:47:27.220 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:47:27.220 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:47:27.291 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:47:27.291 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:47:28.036 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:47:28.139 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:47:28.642 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:47:28.906 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:47:28.907 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:28.907 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:47:28.907 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:47:28.907 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:28.907 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:47:29.113 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados-devel-debuginfo 2024-08-09T14:47:29.655 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:07:07 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:47:29.784 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:47:29.785 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:29.785 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:47:29.785 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:29.785 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:47:29.786 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 109 k 2024-08-09T14:47:29.786 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:29.786 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:47:29.786 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:29.786 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:47:29.786 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:29.787 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 109 k 2024-08-09T14:47:29.787 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 440 k 2024-08-09T14:47:29.787 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:47:30.002 INFO:teuthology.orchestra.run.smithi194.stdout:librados-devel-debuginfo-19.1.0-1345.g0a88632b. 506 kB/s | 109 kB 00:00 2024-08-09T14:47:30.007 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:47:30.007 INFO:teuthology.orchestra.run.smithi194.stdout:Total 503 kB/s | 109 kB 00:00 2024-08-09T14:47:30.007 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:47:30.008 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:47:30.008 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:47:30.041 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:47:30.041 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:47:30.156 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:47:30.266 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-debuginfo-2:19.1.0-1345.g0a88632b.el9 1/1 2024-08-09T14:47:30.624 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados-devel-debuginfo-2:19.1.0-1345.g0a88632b.el9 1/1 2024-08-09T14:47:30.864 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-debuginfo-2:19.1.0-1345.g0a88632b.el9 1/1 2024-08-09T14:47:30.864 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:30.864 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:47:30.864 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:47:30.864 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:30.864 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:47:31.074 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs2-debuginfo 2024-08-09T14:47:31.625 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:07:09 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:47:31.755 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:47:31.756 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:31.756 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:47:31.756 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:31.756 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:47:31.756 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 5.5 M 2024-08-09T14:47:31.756 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:31.756 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:47:31.757 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:31.757 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:47:31.757 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:31.757 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 5.5 M 2024-08-09T14:47:31.757 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 25 M 2024-08-09T14:47:31.757 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:47:32.171 INFO:teuthology.orchestra.run.smithi194.stdout:libcephfs2-debuginfo-19.1.0-1345.g0a88632b.el9. 13 MB/s | 5.5 MB 00:00 2024-08-09T14:47:32.172 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:47:32.172 INFO:teuthology.orchestra.run.smithi194.stdout:Total 13 MB/s | 5.5 MB 00:00 2024-08-09T14:47:32.172 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:47:32.176 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:47:32.176 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:47:32.242 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:47:32.243 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:47:32.938 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:47:33.036 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86 1/1 2024-08-09T14:47:33.515 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86 1/1 2024-08-09T14:47:33.765 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86 1/1 2024-08-09T14:47:33.766 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:33.766 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:47:33.766 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:47:33.766 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:33.766 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:47:33.946 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados2-debuginfo 2024-08-09T14:47:34.492 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:07:12 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:47:34.621 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:47:34.622 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:34.622 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:47:34.622 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:34.622 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:47:34.623 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 46 M 2024-08-09T14:47:34.623 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:34.623 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:47:34.623 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:34.623 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:47:34.623 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:34.623 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 46 M 2024-08-09T14:47:34.623 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 258 M 2024-08-09T14:47:34.623 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:47:36.065 INFO:teuthology.orchestra.run.smithi194.stdout:librados2-debuginfo-19.1.0-1345.g0a88632b.el9.x 32 MB/s | 46 MB 00:01 2024-08-09T14:47:36.066 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:47:36.067 INFO:teuthology.orchestra.run.smithi194.stdout:Total 32 MB/s | 46 MB 00:01 2024-08-09T14:47:36.067 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:47:36.071 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:47:36.071 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:47:36.377 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:47:36.377 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:47:41.545 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:47:41.639 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:47:43.421 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:47:43.662 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:47:43.662 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:43.662 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:47:43.662 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:47:43.663 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:43.663 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:47:43.848 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librbd1-debuginfo 2024-08-09T14:47:44.395 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:07:22 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:47:44.524 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:47:44.524 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:44.525 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:47:44.525 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:44.525 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:47:44.525 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 52 M 2024-08-09T14:47:44.525 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:44.525 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:47:44.525 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:44.525 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:47:44.525 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:44.525 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 52 M 2024-08-09T14:47:44.525 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 310 M 2024-08-09T14:47:44.526 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:47:46.022 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:47:46.126 INFO:teuthology.orchestra.run.smithi194.stdout:librbd1-debuginfo-19.1.0-1345.g0a88632b.el9.x86 33 MB/s | 52 MB 00:01 2024-08-09T14:47:46.127 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:47:46.127 INFO:teuthology.orchestra.run.smithi194.stdout:Total 33 MB/s | 52 MB 00:01 2024-08-09T14:47:46.128 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:47:46.132 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:47:46.132 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:47:46.197 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:47:46.477 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:47:46.477 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:47:52.487 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:47:52.691 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librbd1-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:47:53.733 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:47:53.955 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:47:53.956 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:53.956 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:47:53.956 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:47:53.956 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:53.957 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:47:54.179 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-fuse-debuginfo 2024-08-09T14:47:54.733 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:07:28 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:47:54.850 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:47:54.862 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:47:54.863 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:47:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:47:54.863 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:47:54.863 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:47:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 6.1 M 2024-08-09T14:47:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:54.863 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:47:54.863 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:47:54.863 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:47:54.863 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:54.864 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 6.1 M 2024-08-09T14:47:54.864 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 29 M 2024-08-09T14:47:54.864 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:47:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:47:55.101 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:55.102 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:47:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:47:55.102 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:55.102 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:47:55.285 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-fuse-debuginfo-19.1.0-1345.g0a88632b.el9.x 15 MB/s | 6.1 MB 00:00 2024-08-09T14:47:55.285 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:47:55.286 INFO:teuthology.orchestra.run.smithi155.stdout:Total 14 MB/s | 6.1 MB 00:00 2024-08-09T14:47:55.286 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:47:55.289 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:47:55.290 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:47:55.306 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-cephfs-debuginfo 2024-08-09T14:47:55.360 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:47:55.360 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:47:55.852 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:07:33 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:47:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:47:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:55.983 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:47:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:47:55.983 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 721 k 2024-08-09T14:47:55.983 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:47:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:55.983 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:47:55.984 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:55.984 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 721 k 2024-08-09T14:47:55.984 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 2.4 M 2024-08-09T14:47:55.984 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:47:56.109 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:47:56.209 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:47:56.262 INFO:teuthology.orchestra.run.smithi194.stdout:python3-cephfs-debuginfo-19.1.0-1345.g0a88632b. 2.5 MB/s | 721 kB 00:00 2024-08-09T14:47:56.263 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:47:56.263 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.5 MB/s | 721 kB 00:00 2024-08-09T14:47:56.263 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:47:56.266 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:47:56.266 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:47:56.306 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:47:56.306 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:47:56.480 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:47:56.585 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-debuginfo-2:19.1.0-1345.g0a88632b.el9 1/1 2024-08-09T14:47:56.769 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:47:56.944 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.1.0-1345.g0a88632b.el9 1/1 2024-08-09T14:47:57.006 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:47:57.006 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:57.006 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:47:57.006 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:47:57.006 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:57.006 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:47:57.194 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados-devel-debuginfo 2024-08-09T14:47:57.196 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-debuginfo-2:19.1.0-1345.g0a88632b.el9 1/1 2024-08-09T14:47:57.196 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:57.196 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:47:57.196 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:47:57.196 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:57.196 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:47:57.395 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rados-debuginfo 2024-08-09T14:47:57.739 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:07:31 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:47:57.868 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:47:57.868 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:47:57.869 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:47:57.869 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:47:57.869 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:47:57.869 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 109 k 2024-08-09T14:47:57.869 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:57.869 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:47:57.869 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:47:57.869 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:47:57.869 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:57.869 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 109 k 2024-08-09T14:47:57.869 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 440 k 2024-08-09T14:47:57.869 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:47:57.938 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:07:35 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:47:58.066 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:47:58.067 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:58.067 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:47:58.067 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:58.067 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:47:58.067 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 892 k 2024-08-09T14:47:58.067 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:58.067 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:47:58.067 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:47:58.067 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:47:58.068 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:58.068 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 892 k 2024-08-09T14:47:58.068 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 3.1 M 2024-08-09T14:47:58.068 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:47:58.087 INFO:teuthology.orchestra.run.smithi155.stdout:librados-devel-debuginfo-19.1.0-1345.g0a88632b. 499 kB/s | 109 kB 00:00 2024-08-09T14:47:58.088 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:47:58.088 INFO:teuthology.orchestra.run.smithi155.stdout:Total 496 kB/s | 109 kB 00:00 2024-08-09T14:47:58.089 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:47:58.092 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:47:58.092 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:47:58.127 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:47:58.128 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:47:58.246 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:47:58.348 INFO:teuthology.orchestra.run.smithi194.stdout:python3-rados-debuginfo-19.1.0-1345.g0a88632b.e 3.1 MB/s | 892 kB 00:00 2024-08-09T14:47:58.349 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:47:58.349 INFO:teuthology.orchestra.run.smithi194.stdout:Total 3.1 MB/s | 892 kB 00:00 2024-08-09T14:47:58.349 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:47:58.353 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:47:58.353 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:47:58.361 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-debuginfo-2:19.1.0-1345.g0a88632b.el9 1/1 2024-08-09T14:47:58.392 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:47:58.392 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:47:58.585 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:47:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-debuginfo-2:19.1.0-1345.g0a88632b.el9. 1/1 2024-08-09T14:47:58.703 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados-devel-debuginfo-2:19.1.0-1345.g0a88632b.el9 1/1 2024-08-09T14:47:58.920 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-debuginfo-2:19.1.0-1345.g0a88632b.el9 1/1 2024-08-09T14:47:58.921 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:58.921 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:47:58.921 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:47:58.921 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:58.921 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:47:59.058 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: python3-rados-debuginfo-2:19.1.0-1345.g0a88632b.el9. 1/1 2024-08-09T14:47:59.135 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs2-debuginfo 2024-08-09T14:47:59.302 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-debuginfo-2:19.1.0-1345.g0a88632b.el9. 1/1 2024-08-09T14:47:59.302 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:59.302 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:47:59.302 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:47:59.302 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:47:59.302 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:47:59.503 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rbd-debuginfo 2024-08-09T14:47:59.685 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:07:33 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:47:59.813 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:47:59.814 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:47:59.814 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:47:59.814 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:47:59.814 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:47:59.814 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 5.5 M 2024-08-09T14:47:59.815 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:59.815 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:47:59.815 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:47:59.815 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:47:59.815 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:47:59.815 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 5.5 M 2024-08-09T14:47:59.815 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 25 M 2024-08-09T14:47:59.815 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:48:00.052 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:07:38 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:48:00.183 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:48:00.183 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:00.183 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:48:00.184 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:00.184 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:48:00.184 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 1.3 M 2024-08-09T14:48:00.184 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:00.184 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:48:00.184 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:00.184 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:48:00.184 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:00.184 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 1.3 M 2024-08-09T14:48:00.184 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 5.1 M 2024-08-09T14:48:00.184 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:48:00.241 INFO:teuthology.orchestra.run.smithi155.stdout:libcephfs2-debuginfo-19.1.0-1345.g0a88632b.el9. 13 MB/s | 5.5 MB 00:00 2024-08-09T14:48:00.242 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:00.243 INFO:teuthology.orchestra.run.smithi155.stdout:Total 13 MB/s | 5.5 MB 00:00 2024-08-09T14:48:00.243 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:48:00.246 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:48:00.247 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:48:00.314 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:48:00.314 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:48:00.501 INFO:teuthology.orchestra.run.smithi194.stdout:python3-rbd-debuginfo-19.1.0-1345.g0a88632b.el9 4.2 MB/s | 1.3 MB 00:00 2024-08-09T14:48:00.502 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:00.502 INFO:teuthology.orchestra.run.smithi194.stdout:Total 4.2 MB/s | 1.3 MB 00:00 2024-08-09T14:48:00.503 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:48:00.506 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:48:00.506 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:48:00.549 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:48:00.549 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:48:00.811 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:48:00.925 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:48:01.004 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:48:01.090 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86 1/1 2024-08-09T14:48:01.303 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:48:01.551 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:48:01.552 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:01.552 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:48:01.552 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:01.552 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:01.552 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:48:01.619 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86 1/1 2024-08-09T14:48:01.741 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rgw-debuginfo 2024-08-09T14:48:01.844 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86 1/1 2024-08-09T14:48:01.844 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:01.844 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:48:01.844 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:01.845 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:01.845 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:48:02.066 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librados2-debuginfo 2024-08-09T14:48:02.292 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:07:40 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:48:02.423 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:48:02.424 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:02.424 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:48:02.424 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:02.424 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:48:02.425 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 394 k 2024-08-09T14:48:02.425 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:02.425 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:48:02.425 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:02.425 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:48:02.425 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:02.425 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 394 k 2024-08-09T14:48:02.425 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 1.2 M 2024-08-09T14:48:02.425 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:48:02.617 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:07:36 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:48:02.704 INFO:teuthology.orchestra.run.smithi194.stdout:python3-rgw-debuginfo-19.1.0-1345.g0a88632b.el9 1.4 MB/s | 394 kB 00:00 2024-08-09T14:48:02.704 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:02.705 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.4 MB/s | 394 kB 00:00 2024-08-09T14:48:02.705 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:48:02.708 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:48:02.708 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:48:02.745 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:48:02.745 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:48:02.748 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:48:02.748 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:02.748 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:48:02.749 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:02.749 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:48:02.749 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 46 M 2024-08-09T14:48:02.749 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:02.749 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:48:02.749 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:02.749 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:48:02.749 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:02.750 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 46 M 2024-08-09T14:48:02.750 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 258 M 2024-08-09T14:48:02.750 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:48:02.887 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:48:02.993 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:48:03.317 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:48:03.561 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:48:03.561 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:03.561 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:48:03.561 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:03.561 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:03.561 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:48:03.759 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-fuse-debuginfo 2024-08-09T14:48:04.170 INFO:teuthology.orchestra.run.smithi155.stdout:librados2-debuginfo-19.1.0-1345.g0a88632b.el9.x 32 MB/s | 46 MB 00:01 2024-08-09T14:48:04.171 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:04.171 INFO:teuthology.orchestra.run.smithi155.stdout:Total 32 MB/s | 46 MB 00:01 2024-08-09T14:48:04.171 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:48:04.175 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:48:04.175 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:48:04.312 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:07:42 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:48:04.443 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:48:04.443 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:04.443 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:48:04.443 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:04.443 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:48:04.443 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 516 k 2024-08-09T14:48:04.444 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:04.444 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:48:04.444 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:04.444 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:48:04.444 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:04.444 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 516 k 2024-08-09T14:48:04.444 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 2.6 M 2024-08-09T14:48:04.445 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:48:04.483 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:48:04.484 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:48:04.716 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-fuse-debuginfo-19.1.0-1345.g0a88632b.el9.x8 1.9 MB/s | 516 kB 00:00 2024-08-09T14:48:04.716 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:04.716 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.9 MB/s | 516 kB 00:00 2024-08-09T14:48:04.716 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:48:04.720 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:48:04.720 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:48:04.757 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:48:04.758 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:48:04.923 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:48:05.030 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:48:05.370 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:48:05.624 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:48:05.625 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:05.625 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:48:05.625 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:05.625 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:05.625 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:48:05.832 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-mirror-debuginfo 2024-08-09T14:48:06.382 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:07:44 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:48:06.511 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:48:06.512 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:06.512 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:48:06.513 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:06.513 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:48:06.513 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 52 M 2024-08-09T14:48:06.513 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:06.513 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:48:06.513 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:06.513 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:48:06.513 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:06.513 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 52 M 2024-08-09T14:48:06.513 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 300 M 2024-08-09T14:48:06.513 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:48:08.085 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-mirror-debuginfo-19.1.0-1345.g0a88632b.el9. 33 MB/s | 52 MB 00:01 2024-08-09T14:48:08.086 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:08.086 INFO:teuthology.orchestra.run.smithi194.stdout:Total 33 MB/s | 52 MB 00:01 2024-08-09T14:48:08.086 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:48:08.091 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:48:08.091 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:48:08.429 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:48:08.430 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:48:09.690 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:48:09.792 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:48:11.712 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:48:11.927 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_ 1/1 2024-08-09T14:48:11.927 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:11.927 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:48:11.927 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:11.927 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:11.927 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:48:12.124 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install librbd1-debuginfo 2024-08-09T14:48:12.672 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:07:46 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:48:12.802 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:48:12.802 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:12.802 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:48:12.803 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:12.803 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:48:12.803 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 52 M 2024-08-09T14:48:12.803 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:12.803 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:48:12.803 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:12.803 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:48:12.803 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:12.804 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 52 M 2024-08-09T14:48:12.804 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 310 M 2024-08-09T14:48:12.804 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:48:14.361 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:48:14.423 INFO:teuthology.orchestra.run.smithi155.stdout:librbd1-debuginfo-19.1.0-1345.g0a88632b.el9.x86 32 MB/s | 52 MB 00:01 2024-08-09T14:48:14.424 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:14.424 INFO:teuthology.orchestra.run.smithi155.stdout:Total 32 MB/s | 52 MB 00:01 2024-08-09T14:48:14.424 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:48:14.428 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:48:14.428 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:48:14.466 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86 1/1 2024-08-09T14:48:14.775 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:48:14.776 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:48:16.450 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86 1/1 2024-08-09T14:48:16.699 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86 1/1 2024-08-09T14:48:16.699 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:16.699 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:48:16.699 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:16.699 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:16.700 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:48:16.917 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-nbd-debuginfo 2024-08-09T14:48:17.465 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:07:55 ago on Fri 09 Aug 2024 02:40:22 PM UTC. 2024-08-09T14:48:17.594 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-08-09T14:48:17.594 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:17.594 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-08-09T14:48:17.595 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:17.595 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-08-09T14:48:17.595 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 1.3 M 2024-08-09T14:48:17.595 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:17.595 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-08-09T14:48:17.595 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-08-09T14:48:17.595 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-08-09T14:48:17.595 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:17.595 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 1.3 M 2024-08-09T14:48:17.595 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 5.7 M 2024-08-09T14:48:17.595 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-08-09T14:48:17.912 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-nbd-debuginfo-19.1.0-1345.g0a88632b.el9.x86 4.1 MB/s | 1.3 MB 00:00 2024-08-09T14:48:17.913 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:17.913 INFO:teuthology.orchestra.run.smithi194.stdout:Total 4.0 MB/s | 1.3 MB 00:00 2024-08-09T14:48:17.913 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-08-09T14:48:17.916 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-08-09T14:48:17.916 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-08-09T14:48:17.958 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-08-09T14:48:17.959 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-08-09T14:48:18.198 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-08-09T14:48:18.301 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:48:18.678 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:48:18.924 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:48:18.924 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:18.924 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-08-09T14:48:18.924 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:18.925 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:18.925 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-08-09T14:48:19.112 DEBUG:teuthology.parallel:result is None 2024-08-09T14:48:20.837 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:48:20.937 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librbd1-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:48:23.098 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:48:23.325 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:48:23.325 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:23.325 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:48:23.326 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:23.326 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:23.326 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:48:23.535 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-cephfs-debuginfo 2024-08-09T14:48:24.080 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:07:58 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:48:24.210 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:48:24.211 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:24.211 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:48:24.211 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:24.211 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:48:24.211 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 721 k 2024-08-09T14:48:24.211 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:24.211 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:48:24.211 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:24.211 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:48:24.211 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:24.211 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 721 k 2024-08-09T14:48:24.211 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 2.4 M 2024-08-09T14:48:24.212 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:48:24.520 INFO:teuthology.orchestra.run.smithi155.stdout:python3-cephfs-debuginfo-19.1.0-1345.g0a88632b. 2.3 MB/s | 721 kB 00:00 2024-08-09T14:48:24.521 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:24.521 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.3 MB/s | 721 kB 00:00 2024-08-09T14:48:24.522 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:48:24.525 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:48:24.525 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:48:24.564 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:48:24.564 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:48:24.742 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:48:24.849 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-debuginfo-2:19.1.0-1345.g0a88632b.el9 1/1 2024-08-09T14:48:25.218 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.1.0-1345.g0a88632b.el9 1/1 2024-08-09T14:48:25.433 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-debuginfo-2:19.1.0-1345.g0a88632b.el9 1/1 2024-08-09T14:48:25.433 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:25.433 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:48:25.433 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:25.433 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:25.434 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:48:25.647 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rados-debuginfo 2024-08-09T14:48:26.194 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:08:00 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:48:26.325 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:48:26.325 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:26.325 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:48:26.326 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:26.326 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:48:26.326 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 892 k 2024-08-09T14:48:26.326 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:26.326 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:48:26.326 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:26.326 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:48:26.326 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:26.326 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 892 k 2024-08-09T14:48:26.326 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 3.1 M 2024-08-09T14:48:26.326 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:48:26.721 INFO:teuthology.orchestra.run.smithi155.stdout:python3-rados-debuginfo-19.1.0-1345.g0a88632b.e 2.2 MB/s | 892 kB 00:00 2024-08-09T14:48:26.722 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:26.722 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.2 MB/s | 892 kB 00:00 2024-08-09T14:48:26.722 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:48:26.725 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:48:26.725 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:48:26.765 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:48:26.765 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:48:26.958 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:48:27.060 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-debuginfo-2:19.1.0-1345.g0a88632b.el9. 1/1 2024-08-09T14:48:27.453 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: python3-rados-debuginfo-2:19.1.0-1345.g0a88632b.el9. 1/1 2024-08-09T14:48:27.666 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-debuginfo-2:19.1.0-1345.g0a88632b.el9. 1/1 2024-08-09T14:48:27.666 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:27.666 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:48:27.666 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:27.667 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:27.667 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:48:27.877 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rbd-debuginfo 2024-08-09T14:48:28.422 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:08:02 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:48:28.552 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:48:28.553 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:28.553 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:48:28.553 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:28.553 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:48:28.554 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 1.3 M 2024-08-09T14:48:28.554 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:28.554 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:48:28.554 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:28.554 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:48:28.554 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:28.554 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 1.3 M 2024-08-09T14:48:28.554 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 5.1 M 2024-08-09T14:48:28.554 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:48:29.049 INFO:teuthology.orchestra.run.smithi155.stdout:python3-rbd-debuginfo-19.1.0-1345.g0a88632b.el9 2.7 MB/s | 1.3 MB 00:00 2024-08-09T14:48:29.050 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:29.050 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.7 MB/s | 1.3 MB 00:00 2024-08-09T14:48:29.050 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:48:29.053 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:48:29.053 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:48:29.096 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:48:29.096 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:48:29.339 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:48:29.446 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:48:29.855 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:48:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:48:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:30.074 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:48:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:30.074 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:30.074 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:48:30.271 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rgw-debuginfo 2024-08-09T14:48:30.821 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:08:04 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:48:30.950 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:48:30.951 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:30.951 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:48:30.951 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:30.951 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:48:30.951 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 394 k 2024-08-09T14:48:30.951 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:30.951 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:48:30.951 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:30.951 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:48:30.952 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:30.952 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 394 k 2024-08-09T14:48:30.952 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 1.2 M 2024-08-09T14:48:30.952 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:48:31.436 INFO:teuthology.orchestra.run.smithi155.stdout:python3-rgw-debuginfo-19.1.0-1345.g0a88632b.el9 815 kB/s | 394 kB 00:00 2024-08-09T14:48:31.436 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:31.437 INFO:teuthology.orchestra.run.smithi155.stdout:Total 813 kB/s | 394 kB 00:00 2024-08-09T14:48:31.437 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:48:31.440 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:48:31.440 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:48:31.479 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:48:31.479 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:48:31.621 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:48:31.726 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:48:32.080 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:48:32.307 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x8 1/1 2024-08-09T14:48:32.308 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:32.308 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:48:32.308 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:32.308 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:32.308 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:48:32.511 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-fuse-debuginfo 2024-08-09T14:48:33.060 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:08:07 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:48:33.192 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:48:33.192 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:33.192 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:48:33.192 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:33.192 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:48:33.192 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 516 k 2024-08-09T14:48:33.193 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:33.193 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:48:33.193 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:33.193 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:48:33.193 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:33.193 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 516 k 2024-08-09T14:48:33.193 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 2.6 M 2024-08-09T14:48:33.193 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:48:33.939 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-fuse-debuginfo-19.1.0-1345.g0a88632b.el9.x8 691 kB/s | 516 kB 00:00 2024-08-09T14:48:33.940 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:33.940 INFO:teuthology.orchestra.run.smithi155.stdout:Total 690 kB/s | 516 kB 00:00 2024-08-09T14:48:33.940 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:48:33.944 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:48:33.944 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:48:33.984 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:48:33.984 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:48:34.146 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:48:34.262 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:48:34.610 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:48:34.832 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_6 1/1 2024-08-09T14:48:34.832 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:34.833 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:48:34.833 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:34.833 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:34.833 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:48:35.028 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-mirror-debuginfo 2024-08-09T14:48:35.578 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:08:09 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:48:35.707 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:48:35.707 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:35.707 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:48:35.707 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:35.707 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:48:35.707 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 52 M 2024-08-09T14:48:35.708 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:35.708 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:48:35.708 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:35.708 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:48:35.708 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:35.708 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 52 M 2024-08-09T14:48:35.708 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 300 M 2024-08-09T14:48:35.709 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:48:37.592 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-mirror-debuginfo-19.1.0-1345.g0a88632b.el9. 28 MB/s | 52 MB 00:01 2024-08-09T14:48:37.593 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:37.594 INFO:teuthology.orchestra.run.smithi155.stdout:Total 28 MB/s | 52 MB 00:01 2024-08-09T14:48:37.594 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:48:37.598 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:48:37.598 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:48:37.936 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:48:37.936 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:48:43.886 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:48:43.982 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86 1/1 2024-08-09T14:48:46.098 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86 1/1 2024-08-09T14:48:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86 1/1 2024-08-09T14:48:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:46.321 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:48:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:46.321 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:46.321 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:48:46.538 DEBUG:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-nbd-debuginfo 2024-08-09T14:48:47.090 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:08:21 ago on Fri 09 Aug 2024 02:40:26 PM UTC. 2024-08-09T14:48:47.221 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2024-08-09T14:48:47.222 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:47.222 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2024-08-09T14:48:47.222 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:47.222 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2024-08-09T14:48:47.222 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-debuginfo x86_64 2:19.1.0-1345.g0a88632b.el9 ceph 1.3 M 2024-08-09T14:48:47.222 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:47.222 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2024-08-09T14:48:47.223 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2024-08-09T14:48:47.223 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2024-08-09T14:48:47.223 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:47.223 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 1.3 M 2024-08-09T14:48:47.223 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 5.7 M 2024-08-09T14:48:47.223 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2024-08-09T14:48:47.545 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-nbd-debuginfo-19.1.0-1345.g0a88632b.el9.x86 4.0 MB/s | 1.3 MB 00:00 2024-08-09T14:48:47.546 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2024-08-09T14:48:47.546 INFO:teuthology.orchestra.run.smithi155.stdout:Total 4.0 MB/s | 1.3 MB 00:00 2024-08-09T14:48:47.546 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2024-08-09T14:48:47.549 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2024-08-09T14:48:47.549 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2024-08-09T14:48:47.593 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2024-08-09T14:48:47.593 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2024-08-09T14:48:47.839 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2024-08-09T14:48:47.943 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:48:48.317 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:48:48.538 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 1/1 2024-08-09T14:48:48.539 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:48.539 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2024-08-09T14:48:48.539 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-debuginfo-2:19.1.0-1345.g0a88632b.el9.x86_64 2024-08-09T14:48:48.539 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:48.539 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2024-08-09T14:48:48.732 DEBUG:teuthology.parallel:result is None 2024-08-09T14:48:48.733 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0a88632b45f25ae38493667c7dbd155f6e59b499 2024-08-09T14:48:48.874 DEBUG:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-09T14:48:48.895 INFO:teuthology.orchestra.run.smithi155.stdout:19.1.0-1345.g0a88632b.el9 2024-08-09T14:48:48.896 INFO:teuthology.packaging:The installed version of ceph is 19.1.0-1345.g0a88632b.el9 2024-08-09T14:48:48.896 INFO:teuthology.task.install:The correct ceph version 19.1.0-1345.g0a88632b is installed. 2024-08-09T14:48:48.898 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=0a88632b45f25ae38493667c7dbd155f6e59b499 2024-08-09T14:48:49.035 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-09T14:48:49.061 INFO:teuthology.orchestra.run.smithi194.stdout:19.1.0-1345.g0a88632b.el9 2024-08-09T14:48:49.061 INFO:teuthology.packaging:The installed version of ceph is 19.1.0-1345.g0a88632b.el9 2024-08-09T14:48:49.061 INFO:teuthology.task.install:The correct ceph version 19.1.0-1345.g0a88632b is installed. 2024-08-09T14:48:49.063 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-09T14:48:49.063 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:48:49.064 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-09T14:48:49.089 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:48:49.089 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-09T14:48:49.131 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-09T14:48:49.131 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:48:49.131 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-08-09T14:48:49.156 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-09T14:48:49.221 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:48:49.221 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-08-09T14:48:49.250 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-09T14:48:49.318 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-09T14:48:49.319 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:48:49.319 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-09T14:48:49.343 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-09T14:48:49.408 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:48:49.408 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-09T14:48:49.437 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-09T14:48:49.505 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-09T14:48:49.506 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:48:49.506 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-08-09T14:48:49.530 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-09T14:48:49.596 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:48:49.596 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-08-09T14:48:49.624 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-09T14:48:49.691 INFO:teuthology.run_tasks:Running task ceph... 2024-08-09T14:48:49.769 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-08-09T14:48:49.769 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 777 /var/log/ceph 2024-08-09T14:48:49.772 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /var/log/ceph 2024-08-09T14:48:49.798 INFO:tasks.ceph:Disabling ceph logrotate... 2024-08-09T14:48:49.799 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-09T14:48:49.837 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/logrotate.d/ceph 2024-08-09T14:48:49.865 INFO:tasks.ceph:Creating extra log directories... 2024-08-09T14:48:49.865 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-09T14:48:49.901 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-08-09T14:48:49.941 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-08-09T14:48:49.942 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-08-09T14:48:49.942 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi/7845532', 'branch': 'wip-yuri7-testing-2024-08-08-0804-squid', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7845532', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri7-testing-2024-08-08-0804-squid'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '0a88632b45f25ae38493667c7dbd155f6e59b499', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '0a88632b45f25ae38493667c7dbd155f6e59b499'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri7-testing-2024-08-08-0804-squid', 'sha1': '0a88632b45f25ae38493667c7dbd155f6e59b499'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 2176, 'sha1': '0a88632b45f25ae38493667c7dbd155f6e59b499', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri7-testing-2024-08-08-0804-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '0a88632b45f25ae38493667c7dbd155f6e59b499', 'targets': {'smithi155.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK8ZDYPXlgeQwzp6qxSwh8yTH3mHyBIaCZdSUcRsZDoojfhkT3rTAm4rHXr/Ye+KbmUkOKWA/0/Dm9LwuzWhXDA=', 'smithi194.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKrU4g/VdCy1BITchK7Ie7Ofan1QAueLNbN3w8++rDVpZHWFa1fNDxQp3zRr//NeAnyV2FUIGBAlCdgi8LZZPUg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '43b88056bcf6a0548bdd2d0502e56290cbae52df', 'timestamp': '2024-08-09_13:31:16', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-08-09T14:48:49.942 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-09T14:48:49.966 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-08-09T14:48:49.988 DEBUG:teuthology.orchestra.run.smithi155:> sudo install -d -m0777 -- /var/run/ceph 2024-08-09T14:48:50.021 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m0777 -- /var/run/ceph 2024-08-09T14:48:50.054 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:48:50.054 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/scratch_devs of=/dev/stdout 2024-08-09T14:48:50.100 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-09T14:48:50.100 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_1 2024-08-09T14:48:50.155 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-09T14:48:50.155 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-09T14:48:50.155 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-08-09T14:48:50.155 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-09T14:48:50.155 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-08-09T14:48:50.155 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-08-09 14:42:34.691067877 +0000 2024-08-09T14:48:50.155 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-08-09 14:39:18.809613576 +0000 2024-08-09T14:48:50.155 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-08-09 14:39:18.809613576 +0000 2024-08-09T14:48:50.156 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-08-09 14:39:18.809613576 +0000 2024-08-09T14:48:50.156 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-09T14:48:50.218 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-08-09T14:48:50.219 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-08-09T14:48:50.219 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000207563 s, 2.5 MB/s 2024-08-09T14:48:50.220 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-09T14:48:50.277 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_2 2024-08-09T14:48:50.332 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-09T14:48:50.332 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-09T14:48:50.332 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-08-09T14:48:50.332 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-09T14:48:50.332 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-08-09T14:48:50.333 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-08-09 14:42:34.691067877 +0000 2024-08-09T14:48:50.333 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-08-09 14:39:19.143606665 +0000 2024-08-09T14:48:50.333 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-08-09 14:39:19.143606665 +0000 2024-08-09T14:48:50.333 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-08-09 14:39:19.143606665 +0000 2024-08-09T14:48:50.333 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-09T14:48:50.395 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-08-09T14:48:50.395 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-08-09T14:48:50.395 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000211942 s, 2.4 MB/s 2024-08-09T14:48:50.396 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-09T14:48:50.452 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_3 2024-08-09T14:48:50.507 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-09T14:48:50.508 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-09T14:48:50.508 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-08-09T14:48:50.508 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-09T14:48:50.508 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-08-09T14:48:50.508 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-08-09 14:42:34.691067877 +0000 2024-08-09T14:48:50.508 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-08-09 14:39:19.459600126 +0000 2024-08-09T14:48:50.508 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-08-09 14:39:19.459600126 +0000 2024-08-09T14:48:50.508 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-08-09 14:39:19.459600126 +0000 2024-08-09T14:48:50.508 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-09T14:48:50.571 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-08-09T14:48:50.571 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-08-09T14:48:50.571 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000188829 s, 2.7 MB/s 2024-08-09T14:48:50.573 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-09T14:48:50.629 DEBUG:teuthology.orchestra.run.smithi155:> stat /dev/vg_nvme/lv_4 2024-08-09T14:48:50.684 INFO:teuthology.orchestra.run.smithi155.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-09T14:48:50.684 INFO:teuthology.orchestra.run.smithi155.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-09T14:48:50.684 INFO:teuthology.orchestra.run.smithi155.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-08-09T14:48:50.684 INFO:teuthology.orchestra.run.smithi155.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-09T14:48:50.684 INFO:teuthology.orchestra.run.smithi155.stdout:Context: system_u:object_r:device_t:s0 2024-08-09T14:48:50.684 INFO:teuthology.orchestra.run.smithi155.stdout:Access: 2024-08-09 14:42:34.692067857 +0000 2024-08-09T14:48:50.685 INFO:teuthology.orchestra.run.smithi155.stdout:Modify: 2024-08-09 14:39:19.787593339 +0000 2024-08-09T14:48:50.685 INFO:teuthology.orchestra.run.smithi155.stdout:Change: 2024-08-09 14:39:19.787593339 +0000 2024-08-09T14:48:50.685 INFO:teuthology.orchestra.run.smithi155.stdout: Birth: 2024-08-09 14:39:19.787593339 +0000 2024-08-09T14:48:50.685 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-09T14:48:50.747 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records in 2024-08-09T14:48:50.747 INFO:teuthology.orchestra.run.smithi155.stderr:1+0 records out 2024-08-09T14:48:50.747 INFO:teuthology.orchestra.run.smithi155.stderr:512 bytes copied, 0.000182474 s, 2.8 MB/s 2024-08-09T14:48:50.748 DEBUG:teuthology.orchestra.run.smithi155:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-09T14:48:50.804 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-09T14:48:50.804 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:48:50.805 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-08-09T14:48:50.824 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-09T14:48:50.824 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-08-09T14:48:50.883 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-09T14:48:50.883 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-09T14:48:50.883 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 674 Links: 1 2024-08-09T14:48:50.883 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-09T14:48:50.883 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-08-09T14:48:50.883 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-08-09 14:42:09.135442686 +0000 2024-08-09T14:48:50.883 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-08-09 14:39:19.839719900 +0000 2024-08-09T14:48:50.883 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-08-09 14:39:19.839719900 +0000 2024-08-09T14:48:50.883 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-08-09 14:39:19.839719900 +0000 2024-08-09T14:48:50.884 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-09T14:48:50.950 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-08-09T14:48:50.950 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-08-09T14:48:50.950 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000222631 s, 2.3 MB/s 2024-08-09T14:48:50.952 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-09T14:48:51.011 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-08-09T14:48:51.069 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-09T14:48:51.069 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-09T14:48:51.069 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 693 Links: 1 2024-08-09T14:48:51.069 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-09T14:48:51.069 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-08-09T14:48:51.069 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-08-09 14:42:09.135442686 +0000 2024-08-09T14:48:51.069 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-08-09 14:39:20.179713298 +0000 2024-08-09T14:48:51.069 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-08-09 14:39:20.180713279 +0000 2024-08-09T14:48:51.070 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-08-09 14:39:20.179713298 +0000 2024-08-09T14:48:51.070 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-09T14:48:51.137 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-08-09T14:48:51.137 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-08-09T14:48:51.137 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000210955 s, 2.4 MB/s 2024-08-09T14:48:51.138 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-09T14:48:51.198 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-08-09T14:48:51.255 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-09T14:48:51.255 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-09T14:48:51.255 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 714 Links: 1 2024-08-09T14:48:51.255 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-09T14:48:51.255 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-08-09T14:48:51.255 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-08-09 14:42:09.136442667 +0000 2024-08-09T14:48:51.255 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-08-09 14:39:20.509706891 +0000 2024-08-09T14:48:51.255 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-08-09 14:39:20.509706891 +0000 2024-08-09T14:48:51.256 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-08-09 14:39:20.509706891 +0000 2024-08-09T14:48:51.256 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-09T14:48:51.320 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-08-09T14:48:51.320 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-08-09T14:48:51.320 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000225946 s, 2.3 MB/s 2024-08-09T14:48:51.322 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-09T14:48:51.379 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-08-09T14:48:51.436 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-09T14:48:51.436 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-09T14:48:51.436 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 731 Links: 1 2024-08-09T14:48:51.436 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-09T14:48:51.436 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-08-09T14:48:51.436 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-08-09 14:42:09.136442667 +0000 2024-08-09T14:48:51.436 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-08-09 14:39:20.843700405 +0000 2024-08-09T14:48:51.436 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-08-09 14:39:20.843700405 +0000 2024-08-09T14:48:51.436 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-08-09 14:39:20.843700405 +0000 2024-08-09T14:48:51.437 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-09T14:48:51.501 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-08-09T14:48:51.501 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-08-09T14:48:51.501 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000226548 s, 2.3 MB/s 2024-08-09T14:48:51.502 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-09T14:48:51.560 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-09T14:48:51.560 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi194.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-08-09T14:48:51.560 INFO:tasks.ceph:Generating config... 2024-08-09T14:48:51.561 INFO:tasks.ceph:[client] debug monc = 20 2024-08-09T14:48:51.562 INFO:tasks.ceph:[client] debug ms = 1 2024-08-09T14:48:51.562 INFO:tasks.ceph:[client] debug objecter = 20 2024-08-09T14:48:51.562 INFO:tasks.ceph:[client] debug rados = 20 2024-08-09T14:48:51.562 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-08-09T14:48:51.562 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-08-09T14:48:51.562 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-08-09T14:48:51.562 INFO:tasks.ceph:[global] ms type = async 2024-08-09T14:48:51.562 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-08-09T14:48:51.562 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-08-09T14:48:51.562 INFO:tasks.ceph:[mgr] debug ms = 1 2024-08-09T14:48:51.562 INFO:tasks.ceph:[mon] debug mon = 20 2024-08-09T14:48:51.562 INFO:tasks.ceph:[mon] debug ms = 1 2024-08-09T14:48:51.563 INFO:tasks.ceph:[mon] debug paxos = 20 2024-08-09T14:48:51.563 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-08-09T14:48:51.563 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-08-09T14:48:51.563 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-08-09T14:48:51.563 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-08-09T14:48:51.563 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-08-09T14:48:51.563 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-08-09T14:48:51.563 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-08-09T14:48:51.563 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-08-09T14:48:51.563 INFO:tasks.ceph:[osd] bdev async discard = True 2024-08-09T14:48:51.563 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-08-09T14:48:51.563 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-08-09T14:48:51.563 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-08-09T14:48:51.564 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-08-09T14:48:51.564 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-08-09T14:48:51.564 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-08-09T14:48:51.564 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-08-09T14:48:51.564 INFO:tasks.ceph:[osd] debug monc = 20 2024-08-09T14:48:51.564 INFO:tasks.ceph:[osd] debug ms = 1 2024-08-09T14:48:51.564 INFO:tasks.ceph:[osd] debug osd = 20 2024-08-09T14:48:51.564 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-08-09T14:48:51.564 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-08-09T14:48:51.564 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-08-09T14:48:51.564 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-08-09T14:48:51.564 INFO:tasks.ceph:[osd] osd class default list = * 2024-08-09T14:48:51.564 INFO:tasks.ceph:[osd] osd class load list = * 2024-08-09T14:48:51.564 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-08-09T14:48:51.565 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-08-09T14:48:51.565 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-08-09T14:48:51.565 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-08-09T14:48:51.565 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-08-09T14:48:51.565 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-08-09T14:48:51.565 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-08-09T14:48:51.565 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-08-09T14:48:51.565 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-08-09T14:48:51.565 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-08-09T14:48:51.565 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-08-09T14:48:51.565 INFO:tasks.ceph:Setting up mon.a... 2024-08-09T14:48:51.566 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-08-09T14:48:51.621 INFO:teuthology.orchestra.run.smithi155.stdout:creating /etc/ceph/ceph.keyring 2024-08-09T14:48:51.624 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-08-09T14:48:51.668 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-09T14:48:51.733 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.155'), ('mon.c', '[v2:172.21.15.155:3301,v1:172.21.15.155:6790]'), ('mon.b', '172.21.15.194')] 2024-08-09T14:48:51.734 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.155,[v2:172.21.15.155:3301,v1:172.21.15.155:6790],172.21.15.194', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-08-09T14:48:51.735 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:48:51.735 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-08-09T14:48:51.788 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.155 --addv c '[v2:172.21.15.155:3301,v1:172.21.15.155:6790]' --add b 172.21.15.194 --print /home/ubuntu/cephtest/ceph.monmap 2024-08-09T14:48:51.874 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-08-09T14:48:51.874 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: generated fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:48:51.874 INFO:teuthology.orchestra.run.smithi155.stdout:setting min_mon_release = quincy 2024-08-09T14:48:51.874 INFO:teuthology.orchestra.run.smithi155.stdout:epoch 0 2024-08-09T14:48:51.874 INFO:teuthology.orchestra.run.smithi155.stdout:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:48:51.874 INFO:teuthology.orchestra.run.smithi155.stdout:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:48:51.874 INFO:teuthology.orchestra.run.smithi155.stdout:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:48:51.874 INFO:teuthology.orchestra.run.smithi155.stdout:min_mon_release 17 (quincy) 2024-08-09T14:48:51.874 INFO:teuthology.orchestra.run.smithi155.stdout:election_strategy: 1 2024-08-09T14:48:51.875 INFO:teuthology.orchestra.run.smithi155.stdout:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:48:51.875 INFO:teuthology.orchestra.run.smithi155.stdout:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:48:51.875 INFO:teuthology.orchestra.run.smithi155.stdout:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:48:51.875 INFO:teuthology.orchestra.run.smithi155.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-08-09T14:48:51.876 DEBUG:teuthology.orchestra.run.smithi155:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-08-09T14:48:51.891 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f3d21f4d-627f-4083-bf15-ae5f3fe632de... 2024-08-09T14:48:51.892 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-09T14:48:51.935 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-08-09T14:48:51.975 INFO:teuthology.orchestra.run.smithi155.stdout:[global] 2024-08-09T14:48:51.975 INFO:teuthology.orchestra.run.smithi155.stdout: chdir = "" 2024-08-09T14:48:51.975 INFO:teuthology.orchestra.run.smithi155.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-09T14:48:51.975 INFO:teuthology.orchestra.run.smithi155.stdout: auth supported = cephx 2024-08-09T14:48:51.975 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.975 INFO:teuthology.orchestra.run.smithi155.stdout: filestore xattr use omap = true 2024-08-09T14:48:51.975 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.975 INFO:teuthology.orchestra.run.smithi155.stdout: mon clock drift allowed = 1.000 2024-08-09T14:48:51.975 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.976 INFO:teuthology.orchestra.run.smithi155.stdout: osd crush chooseleaf type = 0 2024-08-09T14:48:51.976 INFO:teuthology.orchestra.run.smithi155.stdout: auth debug = true 2024-08-09T14:48:51.976 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.976 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on old message = true 2024-08-09T14:48:51.976 INFO:teuthology.orchestra.run.smithi155.stdout: ms die on bug = true 2024-08-09T14:48:51.976 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.976 INFO:teuthology.orchestra.run.smithi155.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-09T14:48:51.976 INFO:teuthology.orchestra.run.smithi155.stdout: mon pg warn max object skew = 0 2024-08-09T14:48:51.976 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.976 INFO:teuthology.orchestra.run.smithi155.stdout: # disable pg_autoscaler by default for new pools 2024-08-09T14:48:51.976 INFO:teuthology.orchestra.run.smithi155.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-09T14:48:51.976 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.976 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default size = 2 2024-08-09T14:48:51.977 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow primary affinity = true 2024-08-09T14:48:51.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd allow pg remap = true 2024-08-09T14:48:51.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on legacy crush tunables = false 2024-08-09T14:48:51.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on crush straw calc version zero = false 2024-08-09T14:48:51.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on no sortbitwise = false 2024-08-09T14:48:51.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on osd down out interval zero = false 2024-08-09T14:48:51.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on too few osds = false 2024-08-09T14:48:51.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-09T14:48:51.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-09T14:48:51.977 INFO:teuthology.orchestra.run.smithi155.stdout: mon_allow_pool_size_one = true 2024-08-09T14:48:51.977 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.977 INFO:teuthology.orchestra.run.smithi155.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-09T14:48:51.978 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.978 INFO:teuthology.orchestra.run.smithi155.stdout: osd default data pool replay window = 5 2024-08-09T14:48:51.978 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.978 INFO:teuthology.orchestra.run.smithi155.stdout: mon allow pool delete = true 2024-08-09T14:48:51.978 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.978 INFO:teuthology.orchestra.run.smithi155.stdout: mon cluster log file level = debug 2024-08-09T14:48:51.978 INFO:teuthology.orchestra.run.smithi155.stdout: debug asserts on shutdown = true 2024-08-09T14:48:51.978 INFO:teuthology.orchestra.run.smithi155.stdout: mon health detail to clog = false 2024-08-09T14:48:51.978 INFO:teuthology.orchestra.run.smithi155.stdout: mon host = "172.21.15.155,[v2:172.21.15.155:3301,v1:172.21.15.155:6790],172.21.15.194" 2024-08-09T14:48:51.978 INFO:teuthology.orchestra.run.smithi155.stdout: mon client directed command retry = 5 2024-08-09T14:48:51.978 INFO:teuthology.orchestra.run.smithi155.stdout: mon election default strategy = 1 2024-08-09T14:48:51.978 INFO:teuthology.orchestra.run.smithi155.stdout: ms inject socket failures = 5000 2024-08-09T14:48:51.979 INFO:teuthology.orchestra.run.smithi155.stdout: ms type = async 2024-08-09T14:48:51.979 INFO:teuthology.orchestra.run.smithi155.stdout: osd heartbeat grace = 80 2024-08-09T14:48:51.979 INFO:teuthology.orchestra.run.smithi155.stdout: fsid = f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:48:51.979 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.979 INFO:teuthology.orchestra.run.smithi155.stdout:[osd] 2024-08-09T14:48:51.979 INFO:teuthology.orchestra.run.smithi155.stdout: osd journal size = 100 2024-08-09T14:48:51.979 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.979 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub load threshold = 5.0 2024-08-09T14:48:51.979 INFO:teuthology.orchestra.run.smithi155.stdout: osd scrub max interval = 600 2024-08-09T14:48:51.979 INFO:teuthology.orchestra.run.smithi155.stdout: osd mclock profile = high_recovery_ops 2024-08-09T14:48:51.979 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.979 INFO:teuthology.orchestra.run.smithi155.stdout: osd recover clone overlap = true 2024-08-09T14:48:51.980 INFO:teuthology.orchestra.run.smithi155.stdout: osd recovery max chunk = 1048576 2024-08-09T14:48:51.980 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.980 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug shutdown = true 2024-08-09T14:48:51.980 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug op order = true 2024-08-09T14:48:51.980 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify stray on activate = true 2024-08-09T14:48:51.980 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug trim objects = true 2024-08-09T14:48:51.980 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.980 INFO:teuthology.orchestra.run.smithi155.stdout: osd open classes on start = true 2024-08-09T14:48:51.980 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug pg log writeout = true 2024-08-09T14:48:51.980 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.980 INFO:teuthology.orchestra.run.smithi155.stdout: osd deep scrub update digest min age = 30 2024-08-09T14:48:51.980 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.980 INFO:teuthology.orchestra.run.smithi155.stdout: osd map max advance = 10 2024-08-09T14:48:51.981 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.981 INFO:teuthology.orchestra.run.smithi155.stdout: journal zero on create = true 2024-08-09T14:48:51.981 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.981 INFO:teuthology.orchestra.run.smithi155.stdout: filestore ondisk finisher threads = 3 2024-08-09T14:48:51.981 INFO:teuthology.orchestra.run.smithi155.stdout: filestore apply finisher threads = 3 2024-08-09T14:48:51.981 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.981 INFO:teuthology.orchestra.run.smithi155.stdout: bdev debug aio = true 2024-08-09T14:48:51.981 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug misdirected ops = true 2024-08-09T14:48:51.981 INFO:teuthology.orchestra.run.smithi155.stdout: bdev async discard = True 2024-08-09T14:48:51.981 INFO:teuthology.orchestra.run.smithi155.stdout: bdev enable discard = True 2024-08-09T14:48:51.981 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore allocator = bitmap 2024-08-09T14:48:51.981 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore block size = 96636764160 2024-08-09T14:48:51.982 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore fsck on mount = True 2024-08-09T14:48:51.982 INFO:teuthology.orchestra.run.smithi155.stdout: bluestore zero block detection = True 2024-08-09T14:48:51.982 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluefs = 1 2024-08-09T14:48:51.982 INFO:teuthology.orchestra.run.smithi155.stdout: debug bluestore = 1 2024-08-09T14:48:51.982 INFO:teuthology.orchestra.run.smithi155.stdout: debug monc = 20 2024-08-09T14:48:51.982 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-08-09T14:48:51.982 INFO:teuthology.orchestra.run.smithi155.stdout: debug osd = 20 2024-08-09T14:48:51.982 INFO:teuthology.orchestra.run.smithi155.stdout: debug rocksdb = 10 2024-08-09T14:48:51.982 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd backfillfull_ratio = 0.85 2024-08-09T14:48:51.982 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd full ratio = 0.9 2024-08-09T14:48:51.982 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd nearfull ratio = 0.8 2024-08-09T14:48:51.982 INFO:teuthology.orchestra.run.smithi155.stdout: osd class default list = * 2024-08-09T14:48:51.983 INFO:teuthology.orchestra.run.smithi155.stdout: osd class load list = * 2024-08-09T14:48:51.983 INFO:teuthology.orchestra.run.smithi155.stdout: osd client watch timeout = 120 2024-08-09T14:48:51.983 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify cached snaps = True 2024-08-09T14:48:51.983 INFO:teuthology.orchestra.run.smithi155.stdout: osd debug verify missing on start = True 2024-08-09T14:48:51.983 INFO:teuthology.orchestra.run.smithi155.stdout: osd failsafe full ratio = 0.95 2024-08-09T14:48:51.983 INFO:teuthology.orchestra.run.smithi155.stdout: osd fast shutdown = False 2024-08-09T14:48:51.983 INFO:teuthology.orchestra.run.smithi155.stdout: osd mclock override recovery settings = True 2024-08-09T14:48:51.983 INFO:teuthology.orchestra.run.smithi155.stdout: osd objectstore = bluestore 2024-08-09T14:48:51.983 INFO:teuthology.orchestra.run.smithi155.stdout: osd op queue = debug_random 2024-08-09T14:48:51.983 INFO:teuthology.orchestra.run.smithi155.stdout: osd op queue cut off = debug_random 2024-08-09T14:48:51.983 INFO:teuthology.orchestra.run.smithi155.stdout: osd shutdown pgref assert = True 2024-08-09T14:48:51.983 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.983 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr] 2024-08-09T14:48:51.984 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-08-09T14:48:51.984 INFO:teuthology.orchestra.run.smithi155.stdout: debug mgr = 20 2024-08-09T14:48:51.984 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2024-08-09T14:48:51.984 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2024-08-09T14:48:51.984 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2024-08-09T14:48:51.984 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2024-08-09T14:48:51.984 INFO:teuthology.orchestra.run.smithi155.stdout: mgr/telemetry/nag = false 2024-08-09T14:48:51.984 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.984 INFO:teuthology.orchestra.run.smithi155.stdout:[mon] 2024-08-09T14:48:51.984 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-08-09T14:48:51.984 INFO:teuthology.orchestra.run.smithi155.stdout: debug mon = 20 2024-08-09T14:48:51.984 INFO:teuthology.orchestra.run.smithi155.stdout: debug paxos = 20 2024-08-09T14:48:51.985 INFO:teuthology.orchestra.run.smithi155.stdout: debug auth = 20 2024-08-09T14:48:51.985 INFO:teuthology.orchestra.run.smithi155.stdout: mon data avail warn = 5 2024-08-09T14:48:51.985 INFO:teuthology.orchestra.run.smithi155.stdout: mon mgr mkfs grace = 240 2024-08-09T14:48:51.985 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min pgs per osd = 4 2024-08-09T14:48:51.985 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd reporter subtree level = osd 2024-08-09T14:48:51.985 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd prime pg temp = true 2024-08-09T14:48:51.985 INFO:teuthology.orchestra.run.smithi155.stdout: mon reweight min bytes per osd = 10 2024-08-09T14:48:51.985 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.985 INFO:teuthology.orchestra.run.smithi155.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-09T14:48:51.985 INFO:teuthology.orchestra.run.smithi155.stdout: auth mon ticket ttl = 660 # 11m 2024-08-09T14:48:51.985 INFO:teuthology.orchestra.run.smithi155.stdout: auth service ticket ttl = 240 # 4m 2024-08-09T14:48:51.985 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.985 INFO:teuthology.orchestra.run.smithi155.stdout: # don't complain about insecure global_id in the test suite 2024-08-09T14:48:51.986 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-09T14:48:51.986 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-09T14:48:51.986 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.986 INFO:teuthology.orchestra.run.smithi155.stdout: # 1m isn't quite enough 2024-08-09T14:48:51.986 INFO:teuthology.orchestra.run.smithi155.stdout: mon_down_mkfs_grace = 2m 2024-08-09T14:48:51.986 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.986 INFO:teuthology.orchestra.run.smithi155.stdout: mon_warn_on_filestore_osds = false 2024-08-09T14:48:51.986 INFO:teuthology.orchestra.run.smithi155.stdout: mon min osdmap epochs = 50 2024-08-09T14:48:51.986 INFO:teuthology.orchestra.run.smithi155.stdout: mon osd crush smoke test = False 2024-08-09T14:48:51.986 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune interval = 2 2024-08-09T14:48:51.986 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune min = 15 2024-08-09T14:48:51.986 INFO:teuthology.orchestra.run.smithi155.stdout: mon osdmap full prune txsize = 2 2024-08-09T14:48:51.986 INFO:teuthology.orchestra.run.smithi155.stdout: mon scrub interval = 300 2024-08-09T14:48:51.987 INFO:teuthology.orchestra.run.smithi155.stdout: mon warn on pool no app = False 2024-08-09T14:48:51.987 INFO:teuthology.orchestra.run.smithi155.stdout: paxos service trim min = 10 2024-08-09T14:48:51.987 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:48:51.987 INFO:teuthology.orchestra.run.smithi155.stdout:[client] 2024-08-09T14:48:51.987 INFO:teuthology.orchestra.run.smithi155.stdout: rgw cache enabled = true 2024-08-09T14:48:51.987 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable ops log = true 2024-08-09T14:48:51.987 INFO:teuthology.orchestra.run.smithi155.stdout: rgw enable usage log = true 2024-08-09T14:48:51.987 INFO:teuthology.orchestra.run.smithi155.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-09T14:48:51.987 INFO:teuthology.orchestra.run.smithi155.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-09T14:48:51.987 INFO:teuthology.orchestra.run.smithi155.stdout: debug monc = 20 2024-08-09T14:48:51.987 INFO:teuthology.orchestra.run.smithi155.stdout: debug ms = 1 2024-08-09T14:48:51.987 INFO:teuthology.orchestra.run.smithi155.stdout: debug objecter = 20 2024-08-09T14:48:51.987 INFO:teuthology.orchestra.run.smithi155.stdout: debug rados = 20 2024-08-09T14:48:51.987 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.a] 2024-08-09T14:48:51.988 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.c] 2024-08-09T14:48:51.988 INFO:teuthology.orchestra.run.smithi155.stdout:[mon.b] 2024-08-09T14:48:51.988 INFO:teuthology.orchestra.run.smithi194.stdout:[global] 2024-08-09T14:48:51.988 INFO:teuthology.orchestra.run.smithi194.stdout: chdir = "" 2024-08-09T14:48:51.988 INFO:teuthology.orchestra.run.smithi194.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-08-09T14:48:51.988 INFO:teuthology.orchestra.run.smithi194.stdout: auth supported = cephx 2024-08-09T14:48:51.988 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.989 INFO:teuthology.orchestra.run.smithi194.stdout: filestore xattr use omap = true 2024-08-09T14:48:51.989 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.989 INFO:teuthology.orchestra.run.smithi194.stdout: mon clock drift allowed = 1.000 2024-08-09T14:48:51.989 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.989 INFO:teuthology.orchestra.run.smithi194.stdout: osd crush chooseleaf type = 0 2024-08-09T14:48:51.989 INFO:teuthology.orchestra.run.smithi194.stdout: auth debug = true 2024-08-09T14:48:51.989 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.989 INFO:teuthology.orchestra.run.smithi194.stdout: ms die on old message = true 2024-08-09T14:48:51.989 INFO:teuthology.orchestra.run.smithi194.stdout: ms die on bug = true 2024-08-09T14:48:51.989 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.989 INFO:teuthology.orchestra.run.smithi194.stdout: mon max pg per osd = 10000 # >= luminous 2024-08-09T14:48:51.989 INFO:teuthology.orchestra.run.smithi194.stdout: mon pg warn max object skew = 0 2024-08-09T14:48:51.989 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.989 INFO:teuthology.orchestra.run.smithi194.stdout: # disable pg_autoscaler by default for new pools 2024-08-09T14:48:51.990 INFO:teuthology.orchestra.run.smithi194.stdout: osd_pool_default_pg_autoscale_mode = off 2024-08-09T14:48:51.990 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.990 INFO:teuthology.orchestra.run.smithi194.stdout: osd pool default size = 2 2024-08-09T14:48:51.990 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.990 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd allow primary affinity = true 2024-08-09T14:48:51.990 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd allow pg remap = true 2024-08-09T14:48:51.990 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on legacy crush tunables = false 2024-08-09T14:48:51.990 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on crush straw calc version zero = false 2024-08-09T14:48:51.990 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on no sortbitwise = false 2024-08-09T14:48:51.990 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on osd down out interval zero = false 2024-08-09T14:48:51.990 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on too few osds = false 2024-08-09T14:48:51.990 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-08-09T14:48:51.990 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_pool_no_redundancy = false 2024-08-09T14:48:51.991 INFO:teuthology.orchestra.run.smithi194.stdout: mon_allow_pool_size_one = true 2024-08-09T14:48:51.991 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.991 INFO:teuthology.orchestra.run.smithi194.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-08-09T14:48:51.991 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.991 INFO:teuthology.orchestra.run.smithi194.stdout: osd default data pool replay window = 5 2024-08-09T14:48:51.991 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.991 INFO:teuthology.orchestra.run.smithi194.stdout: mon allow pool delete = true 2024-08-09T14:48:51.991 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.991 INFO:teuthology.orchestra.run.smithi194.stdout: mon cluster log file level = debug 2024-08-09T14:48:51.991 INFO:teuthology.orchestra.run.smithi194.stdout: debug asserts on shutdown = true 2024-08-09T14:48:51.991 INFO:teuthology.orchestra.run.smithi194.stdout: mon health detail to clog = false 2024-08-09T14:48:51.991 INFO:teuthology.orchestra.run.smithi194.stdout: mon host = "172.21.15.155,[v2:172.21.15.155:3301,v1:172.21.15.155:6790],172.21.15.194" 2024-08-09T14:48:51.992 INFO:teuthology.orchestra.run.smithi194.stdout: mon client directed command retry = 5 2024-08-09T14:48:51.992 INFO:teuthology.orchestra.run.smithi194.stdout: mon election default strategy = 1 2024-08-09T14:48:51.992 INFO:teuthology.orchestra.run.smithi194.stdout: ms inject socket failures = 5000 2024-08-09T14:48:51.992 INFO:teuthology.orchestra.run.smithi194.stdout: ms type = async 2024-08-09T14:48:51.992 INFO:teuthology.orchestra.run.smithi194.stdout: osd heartbeat grace = 80 2024-08-09T14:48:51.992 INFO:teuthology.orchestra.run.smithi194.stdout: fsid = f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:48:51.992 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.992 INFO:teuthology.orchestra.run.smithi194.stdout:[osd] 2024-08-09T14:48:51.992 INFO:teuthology.orchestra.run.smithi194.stdout: osd journal size = 100 2024-08-09T14:48:51.992 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.992 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub load threshold = 5.0 2024-08-09T14:48:51.992 INFO:teuthology.orchestra.run.smithi194.stdout: osd scrub max interval = 600 2024-08-09T14:48:51.992 INFO:teuthology.orchestra.run.smithi194.stdout: osd mclock profile = high_recovery_ops 2024-08-09T14:48:51.992 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.993 INFO:teuthology.orchestra.run.smithi194.stdout: osd recover clone overlap = true 2024-08-09T14:48:51.993 INFO:teuthology.orchestra.run.smithi194.stdout: osd recovery max chunk = 1048576 2024-08-09T14:48:51.993 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.993 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug shutdown = true 2024-08-09T14:48:51.993 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug op order = true 2024-08-09T14:48:51.993 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify stray on activate = true 2024-08-09T14:48:51.993 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug trim objects = true 2024-08-09T14:48:51.993 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.993 INFO:teuthology.orchestra.run.smithi194.stdout: osd open classes on start = true 2024-08-09T14:48:51.993 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug pg log writeout = true 2024-08-09T14:48:51.993 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.993 INFO:teuthology.orchestra.run.smithi194.stdout: osd deep scrub update digest min age = 30 2024-08-09T14:48:51.993 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.994 INFO:teuthology.orchestra.run.smithi194.stdout: osd map max advance = 10 2024-08-09T14:48:51.994 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.994 INFO:teuthology.orchestra.run.smithi194.stdout: journal zero on create = true 2024-08-09T14:48:51.994 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.994 INFO:teuthology.orchestra.run.smithi194.stdout: filestore ondisk finisher threads = 3 2024-08-09T14:48:51.994 INFO:teuthology.orchestra.run.smithi194.stdout: filestore apply finisher threads = 3 2024-08-09T14:48:51.994 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.994 INFO:teuthology.orchestra.run.smithi194.stdout: bdev debug aio = true 2024-08-09T14:48:51.994 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug misdirected ops = true 2024-08-09T14:48:51.994 INFO:teuthology.orchestra.run.smithi194.stdout: bdev async discard = True 2024-08-09T14:48:51.994 INFO:teuthology.orchestra.run.smithi194.stdout: bdev enable discard = True 2024-08-09T14:48:51.994 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore allocator = bitmap 2024-08-09T14:48:51.995 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore block size = 96636764160 2024-08-09T14:48:51.995 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore fsck on mount = True 2024-08-09T14:48:51.995 INFO:teuthology.orchestra.run.smithi194.stdout: bluestore zero block detection = True 2024-08-09T14:48:51.995 INFO:teuthology.orchestra.run.smithi194.stdout: debug bluefs = 1 2024-08-09T14:48:51.995 INFO:teuthology.orchestra.run.smithi194.stdout: debug bluestore = 1 2024-08-09T14:48:51.995 INFO:teuthology.orchestra.run.smithi194.stdout: debug monc = 20 2024-08-09T14:48:51.995 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-08-09T14:48:51.995 INFO:teuthology.orchestra.run.smithi194.stdout: debug osd = 20 2024-08-09T14:48:51.995 INFO:teuthology.orchestra.run.smithi194.stdout: debug rocksdb = 10 2024-08-09T14:48:51.995 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd backfillfull_ratio = 0.85 2024-08-09T14:48:51.995 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd full ratio = 0.9 2024-08-09T14:48:51.995 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd nearfull ratio = 0.8 2024-08-09T14:48:51.995 INFO:teuthology.orchestra.run.smithi194.stdout: osd class default list = * 2024-08-09T14:48:51.995 INFO:teuthology.orchestra.run.smithi194.stdout: osd class load list = * 2024-08-09T14:48:51.996 INFO:teuthology.orchestra.run.smithi194.stdout: osd client watch timeout = 120 2024-08-09T14:48:51.996 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify cached snaps = True 2024-08-09T14:48:51.996 INFO:teuthology.orchestra.run.smithi194.stdout: osd debug verify missing on start = True 2024-08-09T14:48:51.996 INFO:teuthology.orchestra.run.smithi194.stdout: osd failsafe full ratio = 0.95 2024-08-09T14:48:51.996 INFO:teuthology.orchestra.run.smithi194.stdout: osd fast shutdown = False 2024-08-09T14:48:51.996 INFO:teuthology.orchestra.run.smithi194.stdout: osd mclock override recovery settings = True 2024-08-09T14:48:51.996 INFO:teuthology.orchestra.run.smithi194.stdout: osd objectstore = bluestore 2024-08-09T14:48:51.996 INFO:teuthology.orchestra.run.smithi194.stdout: osd op queue = debug_random 2024-08-09T14:48:51.996 INFO:teuthology.orchestra.run.smithi194.stdout: osd op queue cut off = debug_random 2024-08-09T14:48:51.996 INFO:teuthology.orchestra.run.smithi194.stdout: osd shutdown pgref assert = True 2024-08-09T14:48:51.996 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.996 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr] 2024-08-09T14:48:51.996 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-08-09T14:48:51.996 INFO:teuthology.orchestra.run.smithi194.stdout: debug mgr = 20 2024-08-09T14:48:51.997 INFO:teuthology.orchestra.run.smithi194.stdout: debug mon = 20 2024-08-09T14:48:51.997 INFO:teuthology.orchestra.run.smithi194.stdout: debug auth = 20 2024-08-09T14:48:51.997 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min pgs per osd = 4 2024-08-09T14:48:51.997 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min bytes per osd = 10 2024-08-09T14:48:51.997 INFO:teuthology.orchestra.run.smithi194.stdout: mgr/telemetry/nag = false 2024-08-09T14:48:51.997 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.997 INFO:teuthology.orchestra.run.smithi194.stdout:[mon] 2024-08-09T14:48:51.997 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-08-09T14:48:51.997 INFO:teuthology.orchestra.run.smithi194.stdout: debug mon = 20 2024-08-09T14:48:51.997 INFO:teuthology.orchestra.run.smithi194.stdout: debug paxos = 20 2024-08-09T14:48:51.997 INFO:teuthology.orchestra.run.smithi194.stdout: debug auth = 20 2024-08-09T14:48:51.997 INFO:teuthology.orchestra.run.smithi194.stdout: mon data avail warn = 5 2024-08-09T14:48:51.998 INFO:teuthology.orchestra.run.smithi194.stdout: mon mgr mkfs grace = 240 2024-08-09T14:48:51.998 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min pgs per osd = 4 2024-08-09T14:48:51.998 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd reporter subtree level = osd 2024-08-09T14:48:51.998 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd prime pg temp = true 2024-08-09T14:48:51.998 INFO:teuthology.orchestra.run.smithi194.stdout: mon reweight min bytes per osd = 10 2024-08-09T14:48:51.998 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.998 INFO:teuthology.orchestra.run.smithi194.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-08-09T14:48:51.998 INFO:teuthology.orchestra.run.smithi194.stdout: auth mon ticket ttl = 660 # 11m 2024-08-09T14:48:51.998 INFO:teuthology.orchestra.run.smithi194.stdout: auth service ticket ttl = 240 # 4m 2024-08-09T14:48:51.998 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.998 INFO:teuthology.orchestra.run.smithi194.stdout: # don't complain about insecure global_id in the test suite 2024-08-09T14:48:51.998 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-08-09T14:48:51.998 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-08-09T14:48:51.998 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.999 INFO:teuthology.orchestra.run.smithi194.stdout: # 1m isn't quite enough 2024-08-09T14:48:51.999 INFO:teuthology.orchestra.run.smithi194.stdout: mon_down_mkfs_grace = 2m 2024-08-09T14:48:51.999 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:51.999 INFO:teuthology.orchestra.run.smithi194.stdout: mon_warn_on_filestore_osds = false 2024-08-09T14:48:51.999 INFO:teuthology.orchestra.run.smithi194.stdout: mon min osdmap epochs = 50 2024-08-09T14:48:51.999 INFO:teuthology.orchestra.run.smithi194.stdout: mon osd crush smoke test = False 2024-08-09T14:48:51.999 INFO:teuthology.orchestra.run.smithi194.stdout: mon osdmap full prune interval = 2 2024-08-09T14:48:51.999 INFO:teuthology.orchestra.run.smithi194.stdout: mon osdmap full prune min = 15 2024-08-09T14:48:51.999 INFO:teuthology.orchestra.run.smithi194.stdout: mon osdmap full prune txsize = 2 2024-08-09T14:48:51.999 INFO:teuthology.orchestra.run.smithi194.stdout: mon scrub interval = 300 2024-08-09T14:48:51.999 INFO:teuthology.orchestra.run.smithi194.stdout: mon warn on pool no app = False 2024-08-09T14:48:51.999 INFO:teuthology.orchestra.run.smithi194.stdout: paxos service trim min = 10 2024-08-09T14:48:51.999 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-08-09T14:48:52.000 INFO:teuthology.orchestra.run.smithi194.stdout:[client] 2024-08-09T14:48:52.000 INFO:teuthology.orchestra.run.smithi194.stdout: rgw cache enabled = true 2024-08-09T14:48:52.000 INFO:teuthology.orchestra.run.smithi194.stdout: rgw enable ops log = true 2024-08-09T14:48:52.000 INFO:teuthology.orchestra.run.smithi194.stdout: rgw enable usage log = true 2024-08-09T14:48:52.000 INFO:teuthology.orchestra.run.smithi194.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-08-09T14:48:52.000 INFO:teuthology.orchestra.run.smithi194.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-08-09T14:48:52.000 INFO:teuthology.orchestra.run.smithi194.stdout: debug monc = 20 2024-08-09T14:48:52.000 INFO:teuthology.orchestra.run.smithi194.stdout: debug ms = 1 2024-08-09T14:48:52.000 INFO:teuthology.orchestra.run.smithi194.stdout: debug objecter = 20 2024-08-09T14:48:52.000 INFO:teuthology.orchestra.run.smithi194.stdout: debug rados = 20 2024-08-09T14:48:52.000 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.a] 2024-08-09T14:48:52.000 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.c] 2024-08-09T14:48:52.000 INFO:teuthology.orchestra.run.smithi194.stdout:[mon.b] 2024-08-09T14:48:52.002 INFO:tasks.ceph:Creating admin key on mon.a... 2024-08-09T14:48:52.002 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-08-09T14:48:52.096 INFO:tasks.ceph:Copying monmap to all nodes... 2024-08-09T14:48:52.097 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:48:52.097 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-08-09T14:48:52.112 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:48:52.112 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-08-09T14:48:52.167 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi155.front.sepia.ceph.com 2024-08-09T14:48:52.168 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:48:52.168 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-09T14:48:52.168 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-09T14:48:52.242 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:48:52.242 DEBUG:teuthology.orchestra.run.smithi155:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-09T14:48:52.298 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi194.front.sepia.ceph.com 2024-08-09T14:48:52.298 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:48:52.298 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.keyring 2024-08-09T14:48:52.298 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-08-09T14:48:52.338 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:48:52.338 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-08-09T14:48:52.395 INFO:tasks.ceph:Setting up mon nodes... 2024-08-09T14:48:52.395 INFO:tasks.ceph:Setting up mgr nodes... 2024-08-09T14:48:52.395 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-08-09T14:48:52.446 INFO:teuthology.orchestra.run.smithi155.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-08-09T14:48:52.449 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-08-09T14:48:52.512 INFO:teuthology.orchestra.run.smithi194.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-08-09T14:48:52.514 INFO:tasks.ceph:Setting up mds nodes... 2024-08-09T14:48:52.515 INFO:tasks.ceph_client:Setting up client nodes... 2024-08-09T14:48:52.515 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-09T14:48:52.556 INFO:teuthology.orchestra.run.smithi155.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-08-09T14:48:52.569 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-08-09T14:48:52.611 INFO:teuthology.orchestra.run.smithi194.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-08-09T14:48:52.624 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-08-09T14:48:52.624 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi155.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi194.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-08-09T14:48:52.624 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-08-09T14:48:52.649 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-09T14:48:52.649 INFO:tasks.ceph:role: osd.0 2024-08-09T14:48:52.650 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com 2024-08-09T14:48:52.650 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-09T14:48:52.722 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-09T14:48:52.722 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-09T14:48:52.722 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-09T14:48:52.722 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-09T14:48:52.723 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-09T14:48:52.723 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-08-09T14:48:52.723 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-09T14:48:52.723 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-09T14:48:52.723 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-09T14:48:52.723 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-09T14:48:53.198 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-08-09T14:48:53.199 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-08-09T14:48:53.200 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-08-09T14:48:53.335 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-08-09T14:48:53.367 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-08-09T14:48:53.432 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-09T14:48:53.432 INFO:tasks.ceph:role: osd.1 2024-08-09T14:48:53.432 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com 2024-08-09T14:48:53.432 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-09T14:48:53.504 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-09T14:48:53.504 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-09T14:48:53.504 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-09T14:48:53.504 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-09T14:48:53.504 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-09T14:48:53.504 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-08-09T14:48:53.504 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-09T14:48:53.504 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-09T14:48:53.504 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-09T14:48:53.504 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-09T14:48:53.964 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-08-09T14:48:53.967 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-08-09T14:48:53.967 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-08-09T14:48:54.048 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-08-09T14:48:54.118 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-08-09T14:48:54.183 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-09T14:48:54.183 INFO:tasks.ceph:role: osd.2 2024-08-09T14:48:54.183 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com 2024-08-09T14:48:54.183 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-09T14:48:54.254 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-09T14:48:54.255 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-09T14:48:54.255 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-09T14:48:54.255 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-09T14:48:54.255 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-09T14:48:54.255 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-08-09T14:48:54.255 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-09T14:48:54.255 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-09T14:48:54.255 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-09T14:48:54.255 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-09T14:48:54.749 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-08-09T14:48:54.751 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-08-09T14:48:54.751 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-08-09T14:48:54.872 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-08-09T14:48:54.902 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-08-09T14:48:54.967 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-08-09T14:48:54.967 INFO:tasks.ceph:role: osd.3 2024-08-09T14:48:54.967 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com 2024-08-09T14:48:54.967 DEBUG:teuthology.orchestra.run.smithi155:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-08-09T14:48:55.040 INFO:teuthology.orchestra.run.smithi155.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-08-09T14:48:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-09T14:48:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-09T14:48:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-09T14:48:55.040 INFO:teuthology.orchestra.run.smithi155.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-09T14:48:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: = sunit=0 swidth=0 blks 2024-08-09T14:48:55.040 INFO:teuthology.orchestra.run.smithi155.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-09T14:48:55.040 INFO:teuthology.orchestra.run.smithi155.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-09T14:48:55.040 INFO:teuthology.orchestra.run.smithi155.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-09T14:48:55.041 INFO:teuthology.orchestra.run.smithi155.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-09T14:48:55.564 INFO:teuthology.orchestra.run.smithi155.stdout:Discarding blocks...Done. 2024-08-09T14:48:55.566 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi155.front.sepia.ceph.com -o noatime 2024-08-09T14:48:55.566 DEBUG:teuthology.orchestra.run.smithi155:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-08-09T14:48:55.692 DEBUG:teuthology.orchestra.run.smithi155:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-08-09T14:48:55.722 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-09T14:48:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:48:55.833+0000 7ff31ec1d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-08-09T14:48:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:48:55.833+0000 7ff31ec1d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-08-09T14:48:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:48:55.833+0000 7ff31ec1d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-08-09T14:48:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:48:55.833+0000 7ff31ec1d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-08-09T14:48:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:48:55.833+0000 7ff31ec1d640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-08-09T14:48:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:48:55.834+0000 7ff31ec1d640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-08-09T14:48:58.678 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-08-09T14:48:58.707 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-09T14:48:58.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:48:58.797+0000 7ff896c30640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-08-09T14:48:58.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:48:58.797+0000 7ff896c30640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-08-09T14:48:58.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:48:58.797+0000 7ff896c30640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-08-09T14:48:58.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:48:58.797+0000 7ff896c30640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-08-09T14:48:58.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:48:58.797+0000 7ff896c30640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-08-09T14:48:58.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:48:58.798+0000 7ff896c30640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-08-09T14:49:01.644 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-08-09T14:49:01.671 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-09T14:49:01.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:01.757+0000 7f343041a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-08-09T14:49:01.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:01.757+0000 7f343041a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-08-09T14:49:01.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:01.757+0000 7f343041a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-08-09T14:49:01.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:01.757+0000 7f343041a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-08-09T14:49:01.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:01.757+0000 7f343041a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-08-09T14:49:01.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:01.758+0000 7f343041a640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-08-09T14:49:04.600 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-08-09T14:49:04.627 DEBUG:teuthology.orchestra.run.smithi155:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-09T14:49:04.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:04.712+0000 7fb121b08640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-08-09T14:49:04.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:04.712+0000 7fb121b08640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-08-09T14:49:04.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:04.712+0000 7fb121b08640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-08-09T14:49:04.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:04.712+0000 7fb121b08640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-08-09T14:49:04.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:04.712+0000 7fb121b08640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-08-09T14:49:04.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:04.712+0000 7fb121b08640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-08-09T14:49:07.557 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-09T14:49:07.587 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-08-09T14:49:07.617 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-09T14:49:07.617 INFO:tasks.ceph:role: osd.4 2024-08-09T14:49:07.617 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi194.front.sepia.ceph.com 2024-08-09T14:49:07.617 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-08-09T14:49:07.692 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-08-09T14:49:07.693 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-09T14:49:07.693 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-09T14:49:07.693 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-09T14:49:07.693 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-09T14:49:07.693 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-08-09T14:49:07.693 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-09T14:49:07.693 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-09T14:49:07.693 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-09T14:49:07.693 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-09T14:49:08.212 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-08-09T14:49:08.214 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-08-09T14:49:08.215 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-08-09T14:49:08.355 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-08-09T14:49:08.389 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-08-09T14:49:08.455 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-09T14:49:08.456 INFO:tasks.ceph:role: osd.5 2024-08-09T14:49:08.456 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi194.front.sepia.ceph.com 2024-08-09T14:49:08.456 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-08-09T14:49:08.530 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-08-09T14:49:08.530 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-09T14:49:08.530 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-09T14:49:08.530 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-09T14:49:08.530 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-09T14:49:08.530 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-08-09T14:49:08.530 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-09T14:49:08.531 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-09T14:49:08.531 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-09T14:49:08.531 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-09T14:49:08.979 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-08-09T14:49:08.982 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-08-09T14:49:08.982 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-08-09T14:49:09.084 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-08-09T14:49:09.117 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-08-09T14:49:09.184 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-09T14:49:09.184 INFO:tasks.ceph:role: osd.6 2024-08-09T14:49:09.184 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi194.front.sepia.ceph.com 2024-08-09T14:49:09.184 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-08-09T14:49:09.257 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-08-09T14:49:09.257 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-09T14:49:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-09T14:49:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-09T14:49:09.258 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-09T14:49:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-08-09T14:49:09.258 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-09T14:49:09.258 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-09T14:49:09.258 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-09T14:49:09.258 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-09T14:49:09.749 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-08-09T14:49:09.752 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-08-09T14:49:09.752 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-08-09T14:49:09.898 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-08-09T14:49:09.930 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-08-09T14:49:09.997 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-08-09T14:49:09.998 INFO:tasks.ceph:role: osd.7 2024-08-09T14:49:09.998 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi194.front.sepia.ceph.com 2024-08-09T14:49:09.998 DEBUG:teuthology.orchestra.run.smithi194:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-08-09T14:49:10.073 INFO:teuthology.orchestra.run.smithi194.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-08-09T14:49:10.073 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 attr=2, projid32bit=1 2024-08-09T14:49:10.073 INFO:teuthology.orchestra.run.smithi194.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-08-09T14:49:10.073 INFO:teuthology.orchestra.run.smithi194.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-08-09T14:49:10.073 INFO:teuthology.orchestra.run.smithi194.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-08-09T14:49:10.073 INFO:teuthology.orchestra.run.smithi194.stdout: = sunit=0 swidth=0 blks 2024-08-09T14:49:10.073 INFO:teuthology.orchestra.run.smithi194.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-08-09T14:49:10.073 INFO:teuthology.orchestra.run.smithi194.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-08-09T14:49:10.073 INFO:teuthology.orchestra.run.smithi194.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-08-09T14:49:10.073 INFO:teuthology.orchestra.run.smithi194.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-08-09T14:49:10.591 INFO:teuthology.orchestra.run.smithi194.stdout:Discarding blocks...Done. 2024-08-09T14:49:10.593 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi194.front.sepia.ceph.com -o noatime 2024-08-09T14:49:10.593 DEBUG:teuthology.orchestra.run.smithi194:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-08-09T14:49:10.726 DEBUG:teuthology.orchestra.run.smithi194:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-08-09T14:49:10.760 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-09T14:49:10.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:10.886+0000 7fb1aa221640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-08-09T14:49:10.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:10.887+0000 7fb1aa221640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-08-09T14:49:10.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:10.887+0000 7fb1aa221640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-08-09T14:49:10.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:10.887+0000 7fb1aa221640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-08-09T14:49:10.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:10.887+0000 7fb1aa221640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-08-09T14:49:10.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:10.887+0000 7fb1aa221640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-08-09T14:49:13.739 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-08-09T14:49:13.768 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-09T14:49:13.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:13.857+0000 7fbb88811640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-08-09T14:49:13.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:13.858+0000 7fbb88811640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-08-09T14:49:13.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:13.858+0000 7fbb88811640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-08-09T14:49:13.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:13.858+0000 7fbb88811640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-08-09T14:49:13.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:13.858+0000 7fbb88811640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-08-09T14:49:13.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:13.858+0000 7fbb88811640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-08-09T14:49:16.703 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-08-09T14:49:16.733 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-09T14:49:16.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:16.822+0000 7f814b56e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-08-09T14:49:16.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:16.823+0000 7f814b56e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-08-09T14:49:16.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:16.823+0000 7f814b56e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-08-09T14:49:16.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:16.823+0000 7f814b56e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-08-09T14:49:16.825 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:16.823+0000 7f814b56e640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-08-09T14:49:16.825 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:16.823+0000 7f814b56e640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-08-09T14:49:19.678 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-08-09T14:49:19.705 DEBUG:teuthology.orchestra.run.smithi194:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-08-09T14:49:19.793 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:19.791+0000 7f7dbb014640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-08-09T14:49:19.793 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:19.791+0000 7f7dbb014640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-08-09T14:49:19.793 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:19.791+0000 7f7dbb014640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-08-09T14:49:19.793 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:19.791+0000 7f7dbb014640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-08-09T14:49:19.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:19.791+0000 7f7dbb014640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-08-09T14:49:19.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:19.792+0000 7f7dbb014640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-08-09T14:49:22.642 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-08-09T14:49:22.671 INFO:tasks.ceph:Reading keys from all nodes... 2024-08-09T14:49:22.671 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:49:22.671 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-08-09T14:49:22.702 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:49:22.702 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-08-09T14:49:22.770 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:49:22.771 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-08-09T14:49:22.839 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:49:22.839 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-08-09T14:49:22.906 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:49:22.906 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-08-09T14:49:22.974 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:49:22.975 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-08-09T14:49:23.004 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:49:23.004 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-08-09T14:49:23.072 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:49:23.073 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-08-09T14:49:23.140 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:49:23.140 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-08-09T14:49:23.207 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:49:23.207 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-08-09T14:49:23.274 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:49:23.274 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-08-09T14:49:23.293 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:49:23.293 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-08-09T14:49:23.330 INFO:tasks.ceph:Adding keys to all mons... 2024-08-09T14:49:23.330 DEBUG:teuthology.orchestra.run.smithi155:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-09T14:49:23.335 DEBUG:teuthology.orchestra.run.smithi194:> sudo tee -a /etc/ceph/ceph.keyring 2024-08-09T14:49:23.374 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.y] 2024-08-09T14:49:23.374 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBULLZm8ceYGhAAqZOAIsqRLwc2Mq6DwHg9sA== 2024-08-09T14:49:23.374 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.0] 2024-08-09T14:49:23.375 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBXLLZmH766MRAAUdAw7opIPfyW15z7FzaQ/g== 2024-08-09T14:49:23.375 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.1] 2024-08-09T14:49:23.375 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBaLLZm2/OSLxAACM8IzF/uDqtf/Lb3cuC6XA== 2024-08-09T14:49:23.375 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.2] 2024-08-09T14:49:23.375 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBdLLZmJ8A3LRAA3WMwpbsrYI81X4Udk795tA== 2024-08-09T14:49:23.375 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.3] 2024-08-09T14:49:23.375 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBgLLZmOziCKhAACT2lAkFuKVbz329VBMpEJQ== 2024-08-09T14:49:23.375 INFO:teuthology.orchestra.run.smithi155.stdout:[mgr.x] 2024-08-09T14:49:23.376 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBULLZmRVmEHhAA3kOG9XGHFAU/xvO0QpWskQ== 2024-08-09T14:49:23.376 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.4] 2024-08-09T14:49:23.376 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBmLLZmWtnqNBAAxASX5BEU+hFWdWV3LOmhwA== 2024-08-09T14:49:23.376 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.5] 2024-08-09T14:49:23.376 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBpLLZmyf4wMxAAuh55aTDhCUdmEDcXpqmeOg== 2024-08-09T14:49:23.376 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.6] 2024-08-09T14:49:23.376 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBsLLZmU+QbMRAAcqagV9wwX8AlGugzJrp/0g== 2024-08-09T14:49:23.376 INFO:teuthology.orchestra.run.smithi155.stdout:[osd.7] 2024-08-09T14:49:23.376 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBvLLZmDaw9LxAAr/abzBxY8Aw9FWHtgvlCvA== 2024-08-09T14:49:23.376 INFO:teuthology.orchestra.run.smithi155.stdout:[client.0] 2024-08-09T14:49:23.376 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBULLZmwn4vIRAAvp3VSI2rkaPDwe1FHuN9Ow== 2024-08-09T14:49:23.376 INFO:teuthology.orchestra.run.smithi155.stdout:[client.1] 2024-08-09T14:49:23.377 INFO:teuthology.orchestra.run.smithi155.stdout: key = AQBULLZmsg91JBAA9LQBtsAZsBi0ZBc0ItGwnQ== 2024-08-09T14:49:23.395 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr.y] 2024-08-09T14:49:23.396 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBULLZm8ceYGhAAqZOAIsqRLwc2Mq6DwHg9sA== 2024-08-09T14:49:23.396 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.0] 2024-08-09T14:49:23.396 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBXLLZmH766MRAAUdAw7opIPfyW15z7FzaQ/g== 2024-08-09T14:49:23.396 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.1] 2024-08-09T14:49:23.396 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBaLLZm2/OSLxAACM8IzF/uDqtf/Lb3cuC6XA== 2024-08-09T14:49:23.396 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.2] 2024-08-09T14:49:23.396 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBdLLZmJ8A3LRAA3WMwpbsrYI81X4Udk795tA== 2024-08-09T14:49:23.396 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.3] 2024-08-09T14:49:23.396 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBgLLZmOziCKhAACT2lAkFuKVbz329VBMpEJQ== 2024-08-09T14:49:23.396 INFO:teuthology.orchestra.run.smithi194.stdout:[mgr.x] 2024-08-09T14:49:23.396 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBULLZmRVmEHhAA3kOG9XGHFAU/xvO0QpWskQ== 2024-08-09T14:49:23.397 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.4] 2024-08-09T14:49:23.397 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBmLLZmWtnqNBAAxASX5BEU+hFWdWV3LOmhwA== 2024-08-09T14:49:23.397 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.5] 2024-08-09T14:49:23.397 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBpLLZmyf4wMxAAuh55aTDhCUdmEDcXpqmeOg== 2024-08-09T14:49:23.397 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.6] 2024-08-09T14:49:23.397 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBsLLZmU+QbMRAAcqagV9wwX8AlGugzJrp/0g== 2024-08-09T14:49:23.397 INFO:teuthology.orchestra.run.smithi194.stdout:[osd.7] 2024-08-09T14:49:23.397 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBvLLZmDaw9LxAAr/abzBxY8Aw9FWHtgvlCvA== 2024-08-09T14:49:23.397 INFO:teuthology.orchestra.run.smithi194.stdout:[client.0] 2024-08-09T14:49:23.397 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBULLZmwn4vIRAAvp3VSI2rkaPDwe1FHuN9Ow== 2024-08-09T14:49:23.397 INFO:teuthology.orchestra.run.smithi194.stdout:[client.1] 2024-08-09T14:49:23.398 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQBULLZmsg91JBAA9LQBtsAZsBi0ZBc0ItGwnQ== 2024-08-09T14:49:23.399 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-09T14:49:23.420 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-09T14:49:23.492 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.495 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.546 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.549 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.599 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.602 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.652 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.654 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.704 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-09T14:49:23.706 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-08-09T14:49:23.762 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.764 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.815 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.817 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.868 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.870 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.920 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.922 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-08-09T14:49:23.973 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-09T14:49:23.975 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-09T14:49:24.028 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-09T14:49:24.030 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-08-09T14:49:24.084 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-08-09T14:49:24.084 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-08-09T14:49:24.109 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-09T14:49:24.204 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-08-09T14:49:24.229 DEBUG:teuthology.orchestra.run.smithi155:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-08-09T14:49:24.293 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-09T14:49:24.391 DEBUG:teuthology.orchestra.run.smithi155:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-08-09T14:49:24.416 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-08-09T14:49:24.441 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-08-09T14:49:24.536 DEBUG:teuthology.orchestra.run.smithi194:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-08-09T14:49:24.560 DEBUG:teuthology.orchestra.run.smithi155:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-09T14:49:24.562 DEBUG:teuthology.orchestra.run.smithi194:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-08-09T14:49:24.615 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-08-09T14:49:24.615 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-08-09T14:49:24.616 INFO:tasks.ceph.mon.a:Restarting daemon 2024-08-09T14:49:24.616 DEBUG:teuthology.orchestra.run.smithi155:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-08-09T14:49:24.618 INFO:tasks.ceph.mon.a:Started 2024-08-09T14:49:24.618 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-08-09T14:49:24.618 INFO:tasks.ceph.mon.c:Restarting daemon 2024-08-09T14:49:24.619 DEBUG:teuthology.orchestra.run.smithi155:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-08-09T14:49:24.620 INFO:tasks.ceph.mon.c:Started 2024-08-09T14:49:24.621 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-08-09T14:49:24.621 INFO:tasks.ceph.mon.b:Restarting daemon 2024-08-09T14:49:24.621 DEBUG:teuthology.orchestra.run.smithi194:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-08-09T14:49:24.657 INFO:tasks.ceph.mon.b:Started 2024-08-09T14:49:24.657 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-08-09T14:49:24.658 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-08-09T14:49:24.658 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-08-09T14:49:24.661 INFO:tasks.ceph.mgr.y:Started 2024-08-09T14:49:24.661 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-08-09T14:49:24.661 DEBUG:teuthology.orchestra.run.smithi194:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-08-09T14:49:24.663 INFO:tasks.ceph.mgr.x:Started 2024-08-09T14:49:24.663 INFO:tasks.ceph:Setting crush tunables to default 2024-08-09T14:49:24.663 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd crush tunables default 2024-08-09T14:49:24.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.762+0000 7fb1ccc13640 10 monclient: get_monmap_and_config 2024-08-09T14:49:24.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.762+0000 7fb1ccc13640 10 monclient: build_initial_monmap 2024-08-09T14:49:24.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.762+0000 7fb1ccc13640 10 monclient: monmap: 2024-08-09T14:49:24.764 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:49:24.764 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:24.764 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:49:24.763203+0000 2024-08-09T14:49:24.764 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:49:24.763203+0000 2024-08-09T14:49:24.764 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:24.764 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:24.764 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:49:24.764 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:49:24.765 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:49:24.765 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:49:24.765 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:49:24.765 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:49:24.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.762+0000 7fb1ccc13640 1 Processor -- start 2024-08-09T14:49:24.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.762+0000 7fb1ccc13640 1 -- start start 2024-08-09T14:49:24.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.762+0000 7fb1ccc13640 10 monclient: init 2024-08-09T14:49:24.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1ccc13640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:24.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1ccc13640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-08-09T14:49:24.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1ccc13640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1c806d470 addr v1:172.21.15.194:6789/0 2024-08-09T14:49:24.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1ccc13640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1c80b44a0 0x7fb1c813ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:24.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1ccc13640 10 monclient(hunting): picked mon.noname-b con 0x7fb1c80b44a0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:24.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1ccc13640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1c80b4e50 addr v1:172.21.15.155:6789/0 2024-08-09T14:49:24.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1ccc13640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1c80b4930 con 0x7fb1c80b4e50 2024-08-09T14:49:24.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1ccc13640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1c80b2e10 con 0x7fb1c806d470 2024-08-09T14:49:24.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1ccc13640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:24.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1ccc13640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fb1c806d850 con 0x7fb1c80b44a0 2024-08-09T14:49:24.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1ccc13640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:24.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1ccc13640 10 monclient(hunting): authenticate will time out at 1171.800623s 2024-08-09T14:49:24.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1c5d9b640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1c80b44a0 msgr2=0x7fb1c813ff10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T14:49:24.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1c6d9d640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fb1c80b4e50 legacy=0x7fb1c814d480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T14:49:24.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1c5d9b640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1c80b44a0 0x7fb1c813ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T14:49:24.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.763+0000 7fb1c659c640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fb1c806d470 legacy=0x7fb1c8146150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T14:49:24.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.964+0000 7fb1c5d9b640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1c80b44a0 msgr2=0x7fb1c813ff10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T14:49:24.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.964+0000 7fb1c6d9d640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fb1c80b4e50 legacy=0x7fb1c814d480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T14:49:24.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.964+0000 7fb1c5d9b640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1c80b44a0 0x7fb1c813ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T14:49:24.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:24.964+0000 7fb1c659c640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fb1c806d470 legacy=0x7fb1c8146150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T14:49:25.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:25.365+0000 7fb1c5d9b640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1c80b44a0 msgr2=0x7fb1c813ff10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T14:49:25.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:25.365+0000 7fb1c6d9d640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fb1c80b4e50 legacy=0x7fb1c814d480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T14:49:25.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:25.365+0000 7fb1c5d9b640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1c80b44a0 0x7fb1c813ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T14:49:25.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:25.365+0000 7fb1c659c640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fb1c806d470 legacy=0x7fb1c8146150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T14:49:26.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:26.166+0000 7fb1c5d9b640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1c80b44a0 msgr2=0x7fb1c813ff10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T14:49:26.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:26.166+0000 7fb1c6d9d640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fb1c80b4e50 legacy=0x7fb1c814d480 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T14:49:26.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:26.166+0000 7fb1c5d9b640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1c80b44a0 0x7fb1c813ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T14:49:26.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:26.166+0000 7fb1c659c640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fb1c806d470 legacy=0x7fb1c8146150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T14:49:27.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 10 monclient(hunting): tick 2024-08-09T14:49:27.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 10 monclient(hunting): _check_auth_tickets 2024-08-09T14:49:27.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 1 monclient(hunting): continuing hunt 2024-08-09T14:49:27.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T14:49:27.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1c80b44a0 msgr2=0x7fb1c813ff10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T14:49:27.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1c80b44a0 0x7fb1c813ff10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:27.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fb1c806d470 legacy=0x7fb1c8146150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T14:49:27.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fb1c80b4e50 legacy=0x7fb1c814d480 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T14:49:27.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 10 monclient: _add_conns ranks=[4,3] 2024-08-09T14:49:27.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb1a80036d0 0x7fb1a8005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:27.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a80036d0 addr v2:172.21.15.194:3300/0 2024-08-09T14:49:27.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb1a80061a0 0x7fb1a8008660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:27.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a80061a0 addr v2:172.21.15.155:3300/0 2024-08-09T14:49:27.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:27.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7fb1c806d850 con 0x7fb1a80061a0 2024-08-09T14:49:27.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:27.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7fb1a8008d10 con 0x7fb1a80036d0 2024-08-09T14:49:27.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1b7fff640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:27.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1c5d9b640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7fb1a80061a0 msgr2=0x7fb1a8008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T14:49:27.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1c5d9b640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb1a80061a0 0x7fb1a8008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T14:49:27.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1c659c640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7fb1a80036d0 msgr2=0x7fb1a8005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T14:49:27.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.763+0000 7fb1c659c640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb1a80036d0 0x7fb1a8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T14:49:27.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.964+0000 7fb1c5d9b640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7fb1a80061a0 msgr2=0x7fb1a8008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T14:49:27.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.964+0000 7fb1c5d9b640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb1a80061a0 0x7fb1a8008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T14:49:27.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.964+0000 7fb1c659c640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7fb1a80036d0 msgr2=0x7fb1a8005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T14:49:27.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:27.964+0000 7fb1c659c640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb1a80036d0 0x7fb1a8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T14:49:28.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:28.365+0000 7fb1c5d9b640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7fb1a80061a0 msgr2=0x7fb1a8008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T14:49:28.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:28.365+0000 7fb1c659c640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7fb1a80036d0 msgr2=0x7fb1a8005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T14:49:28.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:28.365+0000 7fb1c5d9b640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb1a80061a0 0x7fb1a8008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T14:49:28.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:28.365+0000 7fb1c659c640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb1a80036d0 0x7fb1a8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T14:49:29.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:29.166+0000 7fb1c5d9b640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7fb1a80061a0 msgr2=0x7fb1a8008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T14:49:29.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:29.166+0000 7fb1c659c640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7fb1a80036d0 msgr2=0x7fb1a8005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T14:49:29.167 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:29.166+0000 7fb1c5d9b640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb1a80061a0 0x7fb1a8008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T14:49:29.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:29.166+0000 7fb1c659c640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb1a80036d0 0x7fb1a8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T14:49:30.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 10 monclient(hunting): tick 2024-08-09T14:49:30.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 10 monclient(hunting): _check_auth_tickets 2024-08-09T14:49:30.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 1 monclient(hunting): continuing hunt 2024-08-09T14:49:30.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T14:49:30.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7fb1a80036d0 msgr2=0x7fb1a8005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T14:49:30.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb1a80036d0 0x7fb1a8005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:30.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7fb1a80061a0 msgr2=0x7fb1a8008660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T14:49:30.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb1a80061a0 0x7fb1a8008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:30.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1c6d9d640 1 -- reap_dead start 2024-08-09T14:49:30.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-08-09T14:49:30.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a8003b40 0x7fb1a8006610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:30.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a8003b40 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:30.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1a8006b50 addr v1:172.21.15.194:6789/0 2024-08-09T14:49:30.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1a80086d0 addr v1:172.21.15.155:6789/0 2024-08-09T14:49:30.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1c8146860 con 0x7fb1a80086d0 2024-08-09T14:49:30.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1c80b4930 con 0x7fb1a8006b50 2024-08-09T14:49:30.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:30.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fb1c806d850 con 0x7fb1a8003b40 2024-08-09T14:49:30.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.763+0000 7fb1b7fff640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:30.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.764+0000 7fb1c6d9d640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fb1a80086d0 legacy=0x7fb1a800d010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T14:49:30.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.764+0000 7fb1c659c640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a8003b40 msgr2=0x7fb1a8006610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T14:49:30.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.764+0000 7fb1c5d9b640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fb1a8006b50 legacy=0x7fb1a8006fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T14:49:30.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.764+0000 7fb1c659c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a8003b40 0x7fb1a8006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T14:49:30.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.965+0000 7fb1c6d9d640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fb1a80086d0 legacy=0x7fb1a800d010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T14:49:30.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.965+0000 7fb1c659c640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a8003b40 msgr2=0x7fb1a8006610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T14:49:30.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.965+0000 7fb1c5d9b640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fb1a8006b50 legacy=0x7fb1a8006fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T14:49:30.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:30.965+0000 7fb1c659c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a8003b40 0x7fb1a8006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T14:49:31.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:31.366+0000 7fb1c659c640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a8003b40 msgr2=0x7fb1a8006610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T14:49:31.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:31.366+0000 7fb1c6d9d640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fb1a80086d0 legacy=0x7fb1a800d010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T14:49:31.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:31.366+0000 7fb1c659c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a8003b40 0x7fb1a8006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T14:49:31.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:31.366+0000 7fb1c5d9b640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fb1a8006b50 legacy=0x7fb1a8006fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T14:49:31.618 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:00:06.864 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:49:31.928 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:00:07.175 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:49:31.987 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:00:07.230 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:49:31.987 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:00:07.230 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:49:32.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:32.167+0000 7fb1c659c640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a8003b40 msgr2=0x7fb1a8006610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T14:49:32.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:32.167+0000 7fb1c6d9d640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fb1a80086d0 legacy=0x7fb1a800d010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T14:49:32.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:32.167+0000 7fb1c659c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a8003b40 0x7fb1a8006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T14:49:32.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:32.167+0000 7fb1c5d9b640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fb1a8006b50 legacy=0x7fb1a8006fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T14:49:32.302 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:00:07.545 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:49:32.302 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:00:07.546 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:49:33.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 10 monclient(hunting): tick 2024-08-09T14:49:33.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 10 monclient(hunting): _check_auth_tickets 2024-08-09T14:49:33.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 1 monclient(hunting): continuing hunt 2024-08-09T14:49:33.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T14:49:33.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a8003b40 msgr2=0x7fb1a8006610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T14:49:33.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a8003b40 0x7fb1a8006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:33.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7fb1a8006b50 legacy=0x7fb1a8006fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T14:49:33.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7fb1a80086d0 legacy=0x7fb1a800d010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T14:49:33.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 10 monclient: _add_conns ranks=[3,4] 2024-08-09T14:49:33.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 0x7fb1a800ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:33.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a800e8a0 addr v2:172.21.15.155:3300/0 2024-08-09T14:49:33.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:33.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a800f380 addr v2:172.21.15.194:3300/0 2024-08-09T14:49:33.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:33.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7fb1c806d850 con 0x7fb1a800e8a0 2024-08-09T14:49:33.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:33.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7fb1a8008d10 con 0x7fb1a800f380 2024-08-09T14:49:33.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.764+0000 7fb1b7fff640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:33.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.791+0000 7fb1c5d9b640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:33.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.791+0000 7fb1c5d9b640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:42716/0 (socket says 172.21.15.155:42716) 2024-08-09T14:49:33.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.791+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 learned_addr learned my addr 172.21.15.155:0/1567034358 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:49:33.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.791+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request con 0x7fb1a800f380 auth_method 0 2024-08-09T14:49:33.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.791+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:33.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.791+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:33.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.791+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:33.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.792+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 msgr2=0x7fb1a800f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-08-09T14:49:33.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.792+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 msgr2=0x7fb1a800f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:33.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.792+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:33.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.792+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T14:49:33.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.819+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 0x7fb1a800ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:33.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.820+0000 7fb1c659c640 10 monclient(hunting): get_auth_request con 0x7fb1a800e8a0 auth_method 0 2024-08-09T14:49:33.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.820+0000 7fb1c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:33.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.820+0000 7fb1c659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:33.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.820+0000 7fb1c659c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:33.980 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.979+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:33.980 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.979+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:33.981 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.979+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 0x7fb1a800ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:33.981 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.979+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 0x7fb1a800ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T14:49:33.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.993+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:33.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.993+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request con 0x7fb1a800f380 auth_method 0 2024-08-09T14:49:33.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.993+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:33.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.993+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:33.995 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.993+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:33.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.994+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 msgr2=0x7fb1a800f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:33.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.994+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 msgr2=0x7fb1a800f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:33.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.994+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:33.996 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:33.994+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T14:49:34.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.401+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:34.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.402+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request con 0x7fb1a800f380 auth_method 0 2024-08-09T14:49:34.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.402+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:34.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.402+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:34.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.402+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:34.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.413+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 msgr2=0x7fb1a800f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-08-09T14:49:34.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.413+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 msgr2=0x7fb1a800f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:34.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.413+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:34.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.413+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T14:49:34.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.512+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 0x7fb1a800ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:34.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.526+0000 7fb1c659c640 10 monclient(hunting): get_auth_request con 0x7fb1a800e8a0 auth_method 0 2024-08-09T14:49:34.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.526+0000 7fb1c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:34.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.526+0000 7fb1c659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:34.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.526+0000 7fb1c659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:34.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.555+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:34.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.556+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:34.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.556+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 0x7fb1a800ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:34.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.556+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 0x7fb1a800ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T14:49:34.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.989+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 0x7fb1a800ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:34.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.989+0000 7fb1c659c640 10 monclient(hunting): get_auth_request con 0x7fb1a800e8a0 auth_method 0 2024-08-09T14:49:34.990 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.989+0000 7fb1c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:34.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.989+0000 7fb1c659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:34.991 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.989+0000 7fb1c659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:34.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.991+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:34.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.991+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:34.992 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.991+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 0x7fb1a800ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:34.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:34.991+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 0x7fb1a800ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T14:49:35.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.219+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:35.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.220+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request con 0x7fb1a800f380 auth_method 0 2024-08-09T14:49:35.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.220+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:35.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.220+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:35.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.220+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:35.223 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.222+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 msgr2=0x7fb1a800f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:35.223 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.222+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 msgr2=0x7fb1a800f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:35.223 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.222+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:35.223 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.222+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T14:49:35.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.940+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 0x7fb1a800ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:35.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.940+0000 7fb1c659c640 10 monclient(hunting): get_auth_request con 0x7fb1a800e8a0 auth_method 0 2024-08-09T14:49:35.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.940+0000 7fb1c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:35.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.940+0000 7fb1c659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:35.942 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.940+0000 7fb1c659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:35.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.942+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:35.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.942+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:35.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.942+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 0x7fb1a800ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:35.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:35.942+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 0x7fb1a800ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T14:49:36.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 10 monclient(hunting): tick 2024-08-09T14:49:36.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 10 monclient(hunting): _check_auth_tickets 2024-08-09T14:49:36.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 1 monclient(hunting): continuing hunt 2024-08-09T14:49:36.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T14:49:36.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 msgr2=0x7fb1a800f880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T14:49:36.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a800f380 0x7fb1a800f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:36.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T14:49:36.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800e8a0 0x7fb1a800ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:36.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-08-09T14:49:36.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 0x7fb1a8010330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:36.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/1567034358 reap_dead start 2024-08-09T14:49:36.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a800ff50 addr v2:172.21.15.155:3300/0 2024-08-09T14:49:36.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 0x7fb1a8006610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:36.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a800f880 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:36.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 0x7fb1a800ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:36.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a8005b50 addr v2:172.21.15.194:3300/0 2024-08-09T14:49:36.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:36.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 1 -- 172.21.15.155:0/1567034358 --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7fb1c806d850 con 0x7fb1a800ff50 2024-08-09T14:49:36.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:36.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 1 -- 172.21.15.155:0/1567034358 --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7fb1a8008d10 con 0x7fb1a8005b50 2024-08-09T14:49:36.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:36.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 1 -- 172.21.15.155:0/1567034358 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fb1a8006fb0 con 0x7fb1a800f880 2024-08-09T14:49:36.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1b7fff640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:36.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.764+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 0x7fb1a8006610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:36.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.765+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 0x7fb1a8010330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:36.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.765+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request con 0x7fb1a800f880 auth_method 0 2024-08-09T14:49:36.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.765+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:36.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.765+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:36.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.765+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:36.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.765+0000 7fb1c659c640 10 monclient(hunting): get_auth_request con 0x7fb1a800ff50 auth_method 0 2024-08-09T14:49:36.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.765+0000 7fb1c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:36.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.765+0000 7fb1c659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:36.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.765+0000 7fb1c659c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:36.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.766+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 msgr2=0x7fb1a8010330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:36.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.766+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 msgr2=0x7fb1a8010330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:36.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.766+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 0x7fb1a8010330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:36.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.766+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 0x7fb1a8010330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T14:49:36.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.771+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 msgr2=0x7fb1a8006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-08-09T14:49:36.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.771+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 msgr2=0x7fb1a8006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:36.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.771+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 0x7fb1a8006610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:36.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.771+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 0x7fb1a8006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T14:49:36.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.782+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 0x7fb1a800ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:36.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.782+0000 7fb1c6d9d640 10 monclient(hunting): get_auth_request con 0x7fb1a8005b50 auth_method 0 2024-08-09T14:49:36.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.782+0000 7fb1c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:36.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.782+0000 7fb1c6d9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:36.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.782+0000 7fb1c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:36.785 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.783+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-08-09T14:49:36.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.784+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:36.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.784+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 0x7fb1a800ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:36.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.784+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 0x7fb1a800ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T14:49:36.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.967+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 0x7fb1a8010330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:36.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.967+0000 7fb1c659c640 10 monclient(hunting): get_auth_request con 0x7fb1a800ff50 auth_method 0 2024-08-09T14:49:36.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.967+0000 7fb1c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:36.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.967+0000 7fb1c659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:36.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.967+0000 7fb1c659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:36.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.968+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 msgr2=0x7fb1a8010330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:36.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.968+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 msgr2=0x7fb1a8010330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:36.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.968+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 0x7fb1a8010330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:36.970 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.968+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 0x7fb1a8010330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T14:49:36.973 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.972+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 0x7fb1a8006610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:36.973 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.972+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request con 0x7fb1a800f880 auth_method 0 2024-08-09T14:49:36.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.972+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:36.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.972+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:36.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.972+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:36.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.973+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 msgr2=0x7fb1a8006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:36.974 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.973+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 msgr2=0x7fb1a8006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:36.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.973+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 0x7fb1a8006610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:36.975 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.973+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 0x7fb1a8006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T14:49:36.988 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.986+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 0x7fb1a800ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:36.988 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.987+0000 7fb1c6d9d640 10 monclient(hunting): get_auth_request con 0x7fb1a8005b50 auth_method 0 2024-08-09T14:49:36.988 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.987+0000 7fb1c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:36.988 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.987+0000 7fb1c6d9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:36.988 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.987+0000 7fb1c6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:36.989 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.988+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:36.989 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.988+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:36.989 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.988+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 0x7fb1a800ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:36.989 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:36.988+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 0x7fb1a800ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T14:49:37.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.369+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 0x7fb1a8010330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:37.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.370+0000 7fb1c659c640 10 monclient(hunting): get_auth_request con 0x7fb1a800ff50 auth_method 0 2024-08-09T14:49:37.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.370+0000 7fb1c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:37.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.370+0000 7fb1c659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:37.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.370+0000 7fb1c659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:37.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.371+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 msgr2=0x7fb1a8010330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:37.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.371+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 msgr2=0x7fb1a8010330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:37.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.371+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 0x7fb1a8010330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:37.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.371+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 0x7fb1a8010330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T14:49:37.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.374+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 0x7fb1a8006610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:37.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.374+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request con 0x7fb1a800f880 auth_method 0 2024-08-09T14:49:37.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.374+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:37.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.374+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:37.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.374+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:37.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.375+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 msgr2=0x7fb1a8006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:37.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.375+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 msgr2=0x7fb1a8006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:37.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.375+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 0x7fb1a8006610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:37.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.375+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 0x7fb1a8006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T14:49:37.391 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.390+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 0x7fb1a800ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:37.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.391+0000 7fb1c6d9d640 10 monclient(hunting): get_auth_request con 0x7fb1a8005b50 auth_method 0 2024-08-09T14:49:37.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.391+0000 7fb1c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:37.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.391+0000 7fb1c6d9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:37.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.391+0000 7fb1c6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:37.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.392+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:37.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.392+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:37.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.392+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 0x7fb1a800ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:37.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:37.392+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 0x7fb1a800ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T14:49:38.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.172+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 0x7fb1a8010330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:38.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.173+0000 7fb1c659c640 10 monclient(hunting): get_auth_request con 0x7fb1a800ff50 auth_method 0 2024-08-09T14:49:38.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.173+0000 7fb1c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:38.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.173+0000 7fb1c659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:38.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.173+0000 7fb1c659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:38.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.173+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 msgr2=0x7fb1a8010330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:38.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.173+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 msgr2=0x7fb1a8010330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:38.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.173+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 0x7fb1a8010330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:38.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.173+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 0x7fb1a8010330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T14:49:38.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.176+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 0x7fb1a8006610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:38.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.176+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request con 0x7fb1a800f880 auth_method 0 2024-08-09T14:49:38.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.176+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:38.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.176+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:38.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.176+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:38.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.177+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 msgr2=0x7fb1a8006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:38.179 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.177+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 msgr2=0x7fb1a8006610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:38.179 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.177+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 0x7fb1a8006610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:38.179 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.177+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 0x7fb1a8006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T14:49:38.195 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.194+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 0x7fb1a800ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:38.195 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.194+0000 7fb1c6d9d640 10 monclient(hunting): get_auth_request con 0x7fb1a8005b50 auth_method 0 2024-08-09T14:49:38.195 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.194+0000 7fb1c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:38.196 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.194+0000 7fb1c6d9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:38.196 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.194+0000 7fb1c6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:38.196 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.195+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:38.196 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.195+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:38.196 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.195+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 0x7fb1a800ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:38.196 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:38.195+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 0x7fb1a800ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T14:49:39.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 10 monclient(hunting): tick 2024-08-09T14:49:39.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 10 monclient(hunting): _check_auth_tickets 2024-08-09T14:49:39.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 1 monclient(hunting): continuing hunt 2024-08-09T14:49:39.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T14:49:39.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 1 -- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 msgr2=0x7fb1a8006610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T14:49:39.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1a800f880 0x7fb1a8006610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:39.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 msgr2=0x7fb1a800ecb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T14:49:39.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.194:3300/0 conn(0x7fb1a8005b50 0x7fb1a800ecb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:39.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 1 -- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 msgr2=0x7fb1a8010330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T14:49:39.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 1 --2- 172.21.15.155:0/1567034358 >> v2:172.21.15.155:3300/0 conn(0x7fb1a800ff50 0x7fb1a8010330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:39.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 10 monclient: _add_conns ranks=[1,0] 2024-08-09T14:49:39.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb1a80073a0 addr v1:172.21.15.194:6789/0 2024-08-09T14:49:39.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb1a8007f90 addr v1:172.21.15.155:6789/0 2024-08-09T14:49:39.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 1 -- 172.21.15.155:0/1567034358 --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1c80b2e10 con 0x7fb1a8007f90 2024-08-09T14:49:39.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 1 -- 172.21.15.155:0/1567034358 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1c8146860 con 0x7fb1a80073a0 2024-08-09T14:49:39.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:39.764+0000 7fb1b7fff640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:42.158 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.157+0000 7fb1c559a640 1 -- 172.21.15.155:0/1567034358 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3359862910 0 0) 0x7fb1c80b2e10 con 0x7fb1a8007f90 2024-08-09T14:49:42.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.157+0000 7fb1c559a640 10 monclient(hunting): _negotiate 2024-08-09T14:49:42.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.157+0000 7fb1c559a640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:42.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.157+0000 7fb1c559a640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:42.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.157+0000 7fb1c559a640 10 monclient(hunting): my global_id is 4098 2024-08-09T14:49:42.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.157+0000 7fb1c559a640 1 -- 172.21.15.155:0/1567034358 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1b0003520 con 0x7fb1a8007f90 2024-08-09T14:49:42.213 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.210+0000 7fb1c559a640 1 -- 172.21.15.155:0/1567034358 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1632477765 0 0) 0x7fb1b0003520 con 0x7fb1a8007f90 2024-08-09T14:49:42.213 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 1 -- 172.21.15.155:0/1567034358 >> v1:172.21.15.194:6789/0 conn(0x7fb1a80073a0 legacy=0x7fb1a80077b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:42.213 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient: _finish_hunting 0 2024-08-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 1 monclient: found mon.noname-a-legacy 2024-08-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.155:6789/0 2024-08-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 1 -- 172.21.15.155:0/1567034358 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c80b4930 con 0x7fb1a8007f90 2024-08-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient: _finish_auth 0 2024-08-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient: _check_auth_tickets 2024-08-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 1 -- 172.21.15.155:0/1567034358 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1061800462 0 0) 0x7fb1bc009840 con 0x7fb1a8007f90 2024-08-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-08-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient: dump: 2024-08-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] but i'm connected to v1:172.21.15.155:6789/0, reconnecting 2024-08-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:42.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 1 -- 172.21.15.155:0/1567034358 >> v1:172.21.15.155:6789/0 conn(0x7fb1a8007f90 legacy=0x7fb1a800bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:42.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient: _add_conns ranks=[0,1,2] 2024-08-09T14:49:42.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/1567034358 reap_dead start 2024-08-09T14:49:42.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00039e0 0x7fb1b0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:42.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient(hunting): picked mon.a con 0x7fb1b00039e0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:42.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1b0004800 0x7fb1b0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:42.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient(hunting): picked mon.b con 0x7fb1b0004800 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:49:42.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b00053c0 0x7fb1b000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:42.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient(hunting): picked mon.c con 0x7fb1b00053c0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:42.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:42.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 1 -- 172.21.15.155:0/1567034358 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b000c250 con 0x7fb1b00039e0 2024-08-09T14:49:42.217 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:42.217 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 1 -- 172.21.15.155:0/1567034358 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b000c400 con 0x7fb1b0004800 2024-08-09T14:49:42.217 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:42.217 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 1 -- 172.21.15.155:0/1567034358 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fb1b000c600 con 0x7fb1b00053c0 2024-08-09T14:49:42.217 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:42.217 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c559a640 10 monclient(hunting): _finish_auth 0 2024-08-09T14:49:42.217 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.211+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b00053c0 0x7fb1b000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:42.217 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.212+0000 7fb1c6d9d640 10 monclient(hunting): get_auth_request con 0x7fb1b00053c0 auth_method 0 2024-08-09T14:49:42.217 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.212+0000 7fb1c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:42.217 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.212+0000 7fb1c6d9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:42.217 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.212+0000 7fb1c6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:42.218 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.213+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1b0004800 0x7fb1b0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:42.218 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.213+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request con 0x7fb1b0004800 auth_method 0 2024-08-09T14:49:42.218 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.213+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:42.218 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.213+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:42.218 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.213+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:42.220 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.219+0000 7fb1c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:42.220 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.219+0000 7fb1c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:42.220 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.219+0000 7fb1c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-08-09T14:49:42.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.220+0000 7fb1c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:42.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.220+0000 7fb1c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:42.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.220+0000 7fb1c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-08-09T14:49:42.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.229+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00039e0 0x7fb1b0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:42.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.231+0000 7fb1c659c640 10 monclient(hunting): get_auth_request con 0x7fb1b00039e0 auth_method 0 2024-08-09T14:49:42.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.231+0000 7fb1c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:42.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.231+0000 7fb1c659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:42.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.231+0000 7fb1c659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:42.235 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.234+0000 7fb1c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:42.235 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.234+0000 7fb1c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:42.235 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.234+0000 7fb1c659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-08-09T14:49:42.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1c659c640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-08-09T14:49:42.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b00053c0 msgr2=0x7fb1b000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:42.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b00053c0 0x7fb1b000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:42.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1b0004800 msgr2=0x7fb1b0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:42.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1b0004800 0x7fb1b0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:42.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1c659c640 10 monclient: _finish_hunting 0 2024-08-09T14:49:42.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1c659c640 1 monclient: found mon.a 2024-08-09T14:49:42.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1c659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:42.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1c659c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:49:42.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1c659c640 1 -- 172.21.15.155:0/1567034358 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b0003520 con 0x7fb1b00039e0 2024-08-09T14:49:42.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1c659c640 10 monclient: _finish_auth 0 2024-08-09T14:49:42.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1c659c640 10 monclient: _check_auth_tickets 2024-08-09T14:49:42.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1c659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:42.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1ccc13640 5 monclient: authenticate success, global_id 4098 2024-08-09T14:49:42.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.244+0000 7fb1ccc13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:49:42.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.283+0000 7fb1c659c640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00039e0 0x7fb1b0003de0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800f2c0 tx=0x7fb1b0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20c10baa4100fd3 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:42.305 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.304+0000 7fb1c559a640 1 -- 172.21.15.155:0/1567034358 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8039070 con 0x7fb1b00039e0 2024-08-09T14:49:42.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.304+0000 7fb1c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:42.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.304+0000 7fb1c559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:49:42.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.304+0000 7fb1c559a640 10 monclient: dump: 2024-08-09T14:49:42.306 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:49:42.306 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:42.306 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:42.306 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:42.306 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:42.306 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:42.306 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:42.307 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:42.307 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:42.307 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:49:42.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.304+0000 7fb1c559a640 1 -- 172.21.15.155:0/1567034358 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1b8034600 con 0x7fb1b00039e0 2024-08-09T14:49:42.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.304+0000 7fb1c559a640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:42.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.304+0000 7fb1c559a640 1 -- 172.21.15.155:0/1567034358 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8009bc0 con 0x7fb1b00039e0 2024-08-09T14:49:42.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.304+0000 7fb1ccc13640 10 monclient: get_monmap_and_config success 2024-08-09T14:49:42.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.304+0000 7fb1ccc13640 4 set_mon_vals no callback set 2024-08-09T14:49:42.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.304+0000 7fb1c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:42.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.304+0000 7fb1c559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:49:42.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.304+0000 7fb1c559a640 10 monclient: dump: 2024-08-09T14:49:42.307 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:49:42.307 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:42.307 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:42.308 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:42.308 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:42.308 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:42.308 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:42.308 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:42.308 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:42.308 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:49:42.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.305+0000 7fb1ccc13640 10 monclient: shutdown 2024-08-09T14:49:42.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.305+0000 7fb1ccc13640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:42.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.305+0000 7fb1ccc13640 1 -- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00039e0 msgr2=0x7fb1b0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:42.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.305+0000 7fb1ccc13640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00039e0 0x7fb1b0003de0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800f2c0 tx=0x7fb1b0005830 comp rx=0 tx=0).stop 2024-08-09T14:49:42.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.305+0000 7fb1ccc13640 1 -- 172.21.15.155:0/1567034358 shutdown_connections 2024-08-09T14:49:42.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.305+0000 7fb1ccc13640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b00053c0 0x7fb1b000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:42.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.305+0000 7fb1ccc13640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1b0004800 0x7fb1b0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:42.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.305+0000 7fb1ccc13640 1 --2- 172.21.15.155:0/1567034358 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00039e0 0x7fb1b0003de0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:42.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.305+0000 7fb1ccc13640 1 -- 172.21.15.155:0/1567034358 >> 172.21.15.155:0/1567034358 conn(0x7fb1c80b8810 msgr2=0x7fb1c80b8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:42.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.305+0000 7fb1ccc13640 1 -- 172.21.15.155:0/1567034358 shutdown_connections 2024-08-09T14:49:42.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.305+0000 7fb1ccc13640 1 -- 172.21.15.155:0/1567034358 wait complete. 2024-08-09T14:49:42.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.306+0000 7fb1ccc13640 10 monclient: build_initial_monmap 2024-08-09T14:49:42.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.306+0000 7fb1ccc13640 10 monclient: monmap: 2024-08-09T14:49:42.312 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:49:42.313 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:49:42.313 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:49:42.313 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:49:42.313 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:42.313 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:42.313 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:49:42.313 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:49:42.313 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:49:42.313 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:49:42.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.306+0000 7fb1ccc13640 1 librados: starting msgr at 2024-08-09T14:49:42.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.306+0000 7fb1ccc13640 1 librados: starting objecter 2024-08-09T14:49:42.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 1 Processor -- start 2024-08-09T14:49:42.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 1 -- start start 2024-08-09T14:49:42.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 1 librados: setting wanted keys 2024-08-09T14:49:42.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 1 librados: calling monclient init 2024-08-09T14:49:42.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 10 monclient: init 2024-08-09T14:49:42.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:42.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 10 monclient: _add_conns ranks=[1,2,0] 2024-08-09T14:49:42.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1b00039e0 0x7fb1c80acf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:42.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 10 monclient(hunting): picked mon.noname-b con 0x7fb1b00039e0 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:49:42.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b0004800 0x7fb1c8139940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:42.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 10 monclient(hunting): picked mon.noname-c con 0x7fb1b0004800 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:42.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00053c0 0x7fb1c8139e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:42.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 10 monclient(hunting): picked mon.noname-a con 0x7fb1b00053c0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:42.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:42.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c80aad80 con 0x7fb1b00053c0 2024-08-09T14:49:42.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:42.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c80aac00 con 0x7fb1b00039e0 2024-08-09T14:49:42.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:42.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fb1c8146150 con 0x7fb1b0004800 2024-08-09T14:49:42.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:42.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.307+0000 7fb1ccc13640 10 monclient(hunting): authenticate will time out at 1189.344637s 2024-08-09T14:49:42.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.308+0000 7fb1c6d9d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00053c0 0x7fb1c8139e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:42.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.308+0000 7fb1c5d9b640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b0004800 0x7fb1c8139940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:42.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.308+0000 7fb1c5d9b640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b0004800 0x7fb1c8139940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:34046/0 (socket says 172.21.15.155:34046) 2024-08-09T14:49:42.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.308+0000 7fb1c5d9b640 1 -- 172.21.15.155:0/3126878785 learned_addr learned my addr 172.21.15.155:0/3126878785 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:49:42.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.308+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request con 0x7fb1b0004800 auth_method 0 2024-08-09T14:49:42.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.308+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.308+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.308+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.308+0000 7fb1c6d9d640 10 monclient(hunting): get_auth_request con 0x7fb1b00053c0 auth_method 0 2024-08-09T14:49:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.308+0000 7fb1c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.308+0000 7fb1c6d9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.308+0000 7fb1c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.309+0000 7fb1c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.309+0000 7fb1c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.309+0000 7fb1c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.310+0000 7fb1c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.310+0000 7fb1c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:42.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.310+0000 7fb1c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:42.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.319+0000 7fb1c6d9d640 10 monclient(hunting): handle_auth_done global_id 4104 payload 824 2024-08-09T14:49:42.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.319+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b0004800 msgr2=0x7fb1c8139940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:42.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.319+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b0004800 0x7fb1c8139940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:42.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.319+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/3126878785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1b00039e0 msgr2=0x7fb1c80acf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:42.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.319+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1b00039e0 0x7fb1c80acf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:42.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.319+0000 7fb1c6d9d640 10 monclient: _finish_hunting 0 2024-08-09T14:49:42.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.319+0000 7fb1c6d9d640 1 monclient: found mon.noname-a 2024-08-09T14:49:42.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.319+0000 7fb1c6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:42.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.319+0000 7fb1c6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:49:42.322 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.319+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/3126878785 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c80ae750 con 0x7fb1b00053c0 2024-08-09T14:49:42.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.325+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00053c0 msgr2=0x7fb1c8139e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-08-09T14:49:42.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.325+0000 7fb1c6d9d640 1 -- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00053c0 msgr2=0x7fb1c8139e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-08-09T14:49:42.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.325+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00053c0 0x7fb1c8139e80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fb1c80ab4c0 tx=0x7fb1c000b7a0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:42.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.325+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00053c0 0x7fb1c8139e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T14:49:42.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00053c0 0x7fb1c8139e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:42.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1c6d9d640 10 monclient: get_auth_request con 0x7fb1b00053c0 auth_method 0 2024-08-09T14:49:42.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1c6d9d640 0 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00053c0 0x7fb1c8139e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-09T14:49:42.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00053c0 0x7fb1c8139e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:42.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:42.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:42.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 1 -- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00053c0 msgr2=0x7fb1c8139e80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-08-09T14:49:42.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1b00053c0 0x7fb1c8139e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:42.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-08-09T14:49:42.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b0004800 0x7fb1a800ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:42.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 10 monclient(hunting): picked mon.noname-c con 0x7fb1b0004800 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:42.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a8007f60 0x7fb1a8006580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a8007f60 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1b00039e0 0x7fb1a800ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 10 monclient(hunting): picked mon.noname-b con 0x7fb1b00039e0 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:49:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 1 -- 172.21.15.155:0/3126878785 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a80105f0 con 0x7fb1a8007f60 2024-08-09T14:49:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 1 -- 172.21.15.155:0/3126878785 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a8010770 con 0x7fb1b00039e0 2024-08-09T14:49:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 1 -- 172.21.15.155:0/3126878785 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fb1a8007320 con 0x7fb1b0004800 2024-08-09T14:49:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.526+0000 7fb1b6ffd640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.527+0000 7fb1c659c640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b0004800 0x7fb1a800ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:42.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.527+0000 7fb1c6d9d640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1b00039e0 0x7fb1a800ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.527+0000 7fb1c659c640 10 monclient(hunting): get_auth_request con 0x7fb1b0004800 auth_method 0 2024-08-09T14:49:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.527+0000 7fb1c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.527+0000 7fb1c659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.527+0000 7fb1c659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.528+0000 7fb1c6d9d640 10 monclient(hunting): get_auth_request con 0x7fb1b00039e0 auth_method 0 2024-08-09T14:49:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.528+0000 7fb1c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.528+0000 7fb1c6d9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.528+0000 7fb1c6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.528+0000 7fb1c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.528+0000 7fb1c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.528+0000 7fb1c659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-08-09T14:49:42.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.528+0000 7fb1c5d9b640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a8007f60 0x7fb1a8006580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:42.533 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.528+0000 7fb1c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:42.533 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.528+0000 7fb1c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:42.533 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.528+0000 7fb1c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-08-09T14:49:42.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.529+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request con 0x7fb1a8007f60 auth_method 0 2024-08-09T14:49:42.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.529+0000 7fb1c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:42.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.529+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:42.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.529+0000 7fb1c5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:42.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.529+0000 7fb1c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:42.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.529+0000 7fb1c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:42.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.529+0000 7fb1c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-08-09T14:49:42.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.530+0000 7fb1c659c640 10 monclient(hunting): handle_auth_done global_id 4104 payload 293 2024-08-09T14:49:42.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.530+0000 7fb1c659c640 1 -- 172.21.15.155:0/3126878785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1b00039e0 msgr2=0x7fb1a800ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:42.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.530+0000 7fb1c659c640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1b00039e0 0x7fb1a800ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:42.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.530+0000 7fb1c659c640 1 -- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a8007f60 msgr2=0x7fb1a8006580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:42.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.530+0000 7fb1c659c640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a8007f60 0x7fb1a8006580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:42.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.530+0000 7fb1c659c640 10 monclient: _finish_hunting 0 2024-08-09T14:49:42.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.530+0000 7fb1c659c640 1 monclient: found mon.noname-c 2024-08-09T14:49:42.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.530+0000 7fb1c659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:42.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.530+0000 7fb1c659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:49:42.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.530+0000 7fb1c659c640 1 -- 172.21.15.155:0/3126878785 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a8007520 con 0x7fb1b0004800 2024-08-09T14:49:42.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.532+0000 7fb1c659c640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b0004800 0x7fb1a800ecb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800f290 tx=0x7fb1c0011040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:42.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.532+0000 7fb1b6ffd640 10 client.?.objecter ms_handle_connect 0x7fb1b0004800 2024-08-09T14:49:42.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.532+0000 7fb1b6ffd640 10 client.?.objecter resend_mon_ops 2024-08-09T14:49:42.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.539+0000 7fb1b6ffd640 1 -- 172.21.15.155:0/3126878785 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8044020 con 0x7fb1b0004800 2024-08-09T14:49:42.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.539+0000 7fb1b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:42.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.539+0000 7fb1b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:49:42.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.539+0000 7fb1b6ffd640 10 monclient: dump: 2024-08-09T14:49:42.543 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:49:42.543 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:42.543 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:42.543 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:42.544 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:42.544 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:42.544 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:42.544 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:42.544 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:42.544 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:49:42.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.539+0000 7fb1b6ffd640 10 monclient: _finish_auth 0 2024-08-09T14:49:42.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.539+0000 7fb1b6ffd640 10 monclient: _check_auth_tickets 2024-08-09T14:49:42.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.539+0000 7fb1b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:42.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.539+0000 7fb1b6ffd640 1 -- 172.21.15.155:0/3126878785 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1b80347b0 con 0x7fb1b0004800 2024-08-09T14:49:42.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.539+0000 7fb1ccc13640 5 monclient: authenticate success, global_id 4104 2024-08-09T14:49:42.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.539+0000 7fb1ccc13640 10 monclient: _renew_subs 2024-08-09T14:49:42.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.539+0000 7fb1ccc13640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:42.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.539+0000 7fb1ccc13640 1 -- 172.21.15.155:0/3126878785 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c80aea30 con 0x7fb1b0004800 2024-08-09T14:49:42.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.539+0000 7fb1b6ffd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:42.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.540+0000 7fb1b6ffd640 1 -- 172.21.15.155:0/3126878785 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8039040 con 0x7fb1b0004800 2024-08-09T14:49:42.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.540+0000 7fb1b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:42.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.540+0000 7fb1b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:49:42.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.540+0000 7fb1b6ffd640 10 monclient: dump: 2024-08-09T14:49:42.545 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:49:42.545 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:42.545 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:42.545 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:42.545 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:42.545 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:42.545 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:42.546 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:42.546 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:42.546 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:49:42.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.540+0000 7fb1c4d99640 4 set_mon_vals no callback set 2024-08-09T14:49:42.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.540+0000 7fb1ccc13640 10 client.4104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:49:42.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.540+0000 7fb1ccc13640 10 monclient: _renew_subs 2024-08-09T14:49:42.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.540+0000 7fb1ccc13640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:42.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.540+0000 7fb1ccc13640 1 -- 172.21.15.155:0/3126878785 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c80acf20 con 0x7fb1b0004800 2024-08-09T14:49:42.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.540+0000 7fb1ccc13640 1 librados: init done 2024-08-09T14:49:42.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.540+0000 7fb1ccc13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:42.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.540+0000 7fb1ccc13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:42.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.541+0000 7fb1ccc13640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:42.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.541+0000 7fb1ccc13640 1 -- 172.21.15.155:0/3126878785 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1c80b2e10 con 0x7fb1b0004800 2024-08-09T14:49:42.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.547+0000 7fb1b6ffd640 1 -- 172.21.15.155:0/3126878785 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb1b8047070 con 0x7fb1b0004800 2024-08-09T14:49:42.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.547+0000 7fb1b6ffd640 1 -- 172.21.15.155:0/3126878785 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb1b803fa80 con 0x7fb1b0004800 2024-08-09T14:49:42.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.547+0000 7fb1b6ffd640 10 client.4104.objecter ms_dispatch 0x7fb1c8000b90 osd_map(1..1 src has 1..1) 2024-08-09T14:49:42.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.547+0000 7fb1b6ffd640 3 client.4104.objecter handle_osd_map got epochs [1,1] > 0 2024-08-09T14:49:42.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.547+0000 7fb1b6ffd640 3 client.4104.objecter handle_osd_map decoding full epoch 1 2024-08-09T14:49:42.548 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.547+0000 7fb1b6ffd640 20 client.4104.objecter dump_active .. 0 homeless 2024-08-09T14:49:42.678 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:42.676+0000 7f2c8b84e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-09T14:49:42.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.730+0000 7fb1b6ffd640 1 -- 172.21.15.155:0/3126878785 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7fb1b805ef20 con 0x7fb1b0004800 2024-08-09T14:49:42.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.730+0000 7fb1b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:42.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.730+0000 7fb1b6ffd640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:49:42.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.795+0000 7fb1ccc13640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-08-09T14:49:42.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.795+0000 7fb1ccc13640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-08-09T14:49:42.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.795+0000 7fb1ccc13640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:42.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:42.795+0000 7fb1ccc13640 1 -- 172.21.15.155:0/3126878785 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fb1c813e8e0 con 0x7fb1b0004800 2024-08-09T14:49:42.840 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:42.839+0000 7f2c8b84e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-09T14:49:42.844 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:42.843+0000 7f33aaa46040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-08-09T14:49:42.893 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:42.892+0000 7f2c8b84e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-09T14:49:43.014 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:43.013+0000 7f33aaa46040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-08-09T14:49:43.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.040+0000 7fb1b6ffd640 1 -- 172.21.15.155:0/3126878785 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fb1b804d090 con 0x7fb1b0004800 2024-08-09T14:49:43.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.040+0000 7fb1b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-08-09T14:49:43.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.040+0000 7fb1b6ffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-08-09T14:49:43.042 INFO:teuthology.orchestra.run.smithi155.stderr:adjusted tunables profile to default 2024-08-09T14:49:43.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 10 librados: watch_flush enter 2024-08-09T14:49:43.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 10 librados: watch_flush exit 2024-08-09T14:49:43.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 20 client.4104.objecter shutdown clearing up homeless session... 2024-08-09T14:49:43.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 10 client.4104.objecter successfully canceled tick 2024-08-09T14:49:43.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 10 monclient: shutdown 2024-08-09T14:49:43.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:43.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 1 -- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b0004800 msgr2=0x7fb1a800ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:43.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b0004800 0x7fb1a800ecb0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb1a800f290 tx=0x7fb1c0011040 comp rx=0 tx=0).stop 2024-08-09T14:49:43.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 1 -- 172.21.15.155:0/3126878785 shutdown_connections 2024-08-09T14:49:43.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb1b00039e0 0x7fb1a800ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:43.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb1a8007f60 0x7fb1a8006580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:43.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 1 --2- 172.21.15.155:0/3126878785 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb1b0004800 0x7fb1a800ecb0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:43.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 1 -- 172.21.15.155:0/3126878785 >> 172.21.15.155:0/3126878785 conn(0x7fb1c80b8810 msgr2=0x7fb1c813daa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:43.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 1 -- 172.21.15.155:0/3126878785 shutdown_connections 2024-08-09T14:49:43.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 1 -- 172.21.15.155:0/3126878785 wait complete. 2024-08-09T14:49:43.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:43.041+0000 7fb1ccc13640 1 librados: shutdown 2024-08-09T14:49:43.054 INFO:tasks.ceph:check_enable_crimson: False 2024-08-09T14:49:43.054 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-08-09T14:49:43.055 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:49:43.055 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-08-09T14:49:43.068 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:43.067+0000 7f33aaa46040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-08-09T14:49:43.080 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:49:43.080 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-08-09T14:49:43.149 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:49:43.149 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-08-09T14:49:43.215 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:49:43.215 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-08-09T14:49:43.239 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:43.238+0000 7f2c8b84e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-09T14:49:43.263 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:49:43.263 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-08-09T14:49:43.287 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:43.286+0000 7f2c8b84e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-09T14:49:43.289 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:49:43.289 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-08-09T14:49:43.325 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:43.324+0000 7f33aaa46040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-08-09T14:49:43.349 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:49:43.349 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-08-09T14:49:43.373 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:43.371+0000 7f33aaa46040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-08-09T14:49:43.397 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-08-09T14:49:43.397 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-08-09T14:49:43.461 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 6526d474-661b-48e6-bcc7-6288c7477f78 0 2024-08-09T14:49:43.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.600+0000 7f43b560a640 10 monclient: get_monmap_and_config 2024-08-09T14:49:43.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.600+0000 7f43b560a640 10 monclient: build_initial_monmap 2024-08-09T14:49:43.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.600+0000 7f43b560a640 10 monclient: monmap: 2024-08-09T14:49:43.602 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:43.602 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:43.603 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:49:43.601364+0000 2024-08-09T14:49:43.603 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:49:43.601364+0000 2024-08-09T14:49:43.603 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:43.603 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:43.603 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:49:43.603 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:49:43.603 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:49:43.603 INFO:teuthology.orchestra.run.smithi194.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:49:43.603 INFO:teuthology.orchestra.run.smithi194.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:49:43.603 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:43.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 1 Processor -- start 2024-08-09T14:49:43.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 1 -- start start 2024-08-09T14:49:43.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 10 monclient: init 2024-08-09T14:49:43.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:43.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-08-09T14:49:43.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f43b00b00f0 0x7f43b013bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:43.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 10 monclient(hunting): picked mon.noname-a con 0x7f43b00b00f0 addr v2:172.21.15.155:3300/0 2024-08-09T14:49:43.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f43b013c450 0x7f43b013ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:43.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 10 monclient(hunting): picked mon.noname-c con 0x7f43b013c450 addr v2:172.21.15.194:3300/0 2024-08-09T14:49:43.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43b013f4c0 0x7f43b0142500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:43.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 10 monclient(hunting): picked mon.noname-b con 0x7f43b013f4c0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:43.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:43.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f43b00b04d0 con 0x7f43b00b00f0 2024-08-09T14:49:43.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:43.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f43b013c8a0 con 0x7f43b013c450 2024-08-09T14:49:43.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:43.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f43b013f950 con 0x7f43b013f4c0 2024-08-09T14:49:43.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:43.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.601+0000 7f43b560a640 10 monclient(hunting): authenticate will time out at 1208.635731s 2024-08-09T14:49:43.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.602+0000 7f43ae7fc640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f43b013c450 0x7f43b013ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:43.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.602+0000 7f43aeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f43b00b00f0 0x7f43b013bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:43.607 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.602+0000 7f43ae7fc640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f43b013c450 0x7f43b013ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:46278/0 (socket says 172.21.15.194:46278) 2024-08-09T14:49:43.607 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.602+0000 7f43ae7fc640 1 -- 172.21.15.194:0/2482927076 learned_addr learned my addr 172.21.15.194:0/2482927076 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-08-09T14:49:43.607 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.602+0000 7f43ae7fc640 10 monclient(hunting): get_auth_request con 0x7f43b013c450 auth_method 0 2024-08-09T14:49:43.607 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.602+0000 7f43ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:43.607 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.602+0000 7f43ae7fc640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:43.607 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.602+0000 7f43ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:43.607 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.603+0000 7f43aeffd640 10 monclient(hunting): get_auth_request con 0x7f43b00b00f0 auth_method 0 2024-08-09T14:49:43.607 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.603+0000 7f43aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:43.607 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.603+0000 7f43aeffd640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:43.607 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.603+0000 7f43aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:43.607 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.603+0000 7f43ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:43.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.603+0000 7f43ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:43.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.603+0000 7f43ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:43.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.604+0000 7f43ae7fc640 10 monclient(hunting): handle_auth_done global_id 4105 payload 274 2024-08-09T14:49:43.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.604+0000 7f43ae7fc640 1 -- 172.21.15.194:0/2482927076 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43b013f4c0 msgr2=0x7f43b0142500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:43.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.604+0000 7f43ae7fc640 1 --2- 172.21.15.194:0/2482927076 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43b013f4c0 0x7f43b0142500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:43.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.604+0000 7f43ae7fc640 1 -- 172.21.15.194:0/2482927076 >> v2:172.21.15.155:3300/0 conn(0x7f43b00b00f0 msgr2=0x7f43b013bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:43.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.604+0000 7f43ae7fc640 1 --2- 172.21.15.194:0/2482927076 >> v2:172.21.15.155:3300/0 conn(0x7f43b00b00f0 0x7f43b013bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:43.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.604+0000 7f43ae7fc640 10 monclient: _finish_hunting 0 2024-08-09T14:49:43.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.604+0000 7f43ae7fc640 1 monclient: found mon.noname-c 2024-08-09T14:49:43.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.604+0000 7f43ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:43.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.604+0000 7f43ae7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:49:43.610 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.604+0000 7f43ae7fc640 1 -- 172.21.15.194:0/2482927076 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b00ad7b0 con 0x7f43b013c450 2024-08-09T14:49:43.610 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.606+0000 7f43ae7fc640 1 --2- 172.21.15.194:0/2482927076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43b013c450 0x7f43b013ef80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f43b00acdc0 tx=0x7f43a402cdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=242d8ba743a17bae server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:43.611 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43adffb640 1 -- 172.21.15.194:0/2482927076 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a403a070 con 0x7f43b013c450 2024-08-09T14:49:43.611 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:43.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43adffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:49:43.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43adffb640 10 monclient: dump: 2024-08-09T14:49:43.612 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:43.612 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:43.612 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:43.612 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:43.612 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:43.612 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:43.612 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:43.612 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:43.612 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:43.612 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:43.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43adffb640 10 monclient: _finish_auth 0 2024-08-09T14:49:43.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43adffb640 10 monclient: _check_auth_tickets 2024-08-09T14:49:43.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:43.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43adffb640 1 -- 172.21.15.194:0/2482927076 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a402dba0 con 0x7f43b013c450 2024-08-09T14:49:43.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43b560a640 5 monclient: authenticate success, global_id 4105 2024-08-09T14:49:43.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43b560a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:49:43.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43adffb640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:43.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43b560a640 10 monclient: get_monmap_and_config success 2024-08-09T14:49:43.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43b560a640 4 set_mon_vals no callback set 2024-08-09T14:49:43.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43adffb640 1 -- 172.21.15.194:0/2482927076 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a400b040 con 0x7f43b013c450 2024-08-09T14:49:43.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:43.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:49:43.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.610+0000 7f43adffb640 10 monclient: dump: 2024-08-09T14:49:43.614 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:43.614 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:43.614 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:43.614 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:43.614 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:43.614 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:43.614 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:43.614 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:43.614 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:43.615 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:43.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.611+0000 7f43b560a640 10 monclient: shutdown 2024-08-09T14:49:43.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.611+0000 7f43b560a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:43.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.611+0000 7f43b560a640 1 -- 172.21.15.194:0/2482927076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43b013c450 msgr2=0x7f43b013ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:43.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.611+0000 7f43b560a640 1 --2- 172.21.15.194:0/2482927076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43b013c450 0x7f43b013ef80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f43b00acdc0 tx=0x7f43a402cdf0 comp rx=0 tx=0).stop 2024-08-09T14:49:43.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.611+0000 7f43b560a640 1 -- 172.21.15.194:0/2482927076 shutdown_connections 2024-08-09T14:49:43.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.611+0000 7f43b560a640 1 --2- 172.21.15.194:0/2482927076 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43b013f4c0 0x7f43b0142500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:43.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.611+0000 7f43b560a640 1 --2- 172.21.15.194:0/2482927076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43b013c450 0x7f43b013ef80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:43.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.611+0000 7f43b560a640 1 --2- 172.21.15.194:0/2482927076 >> v2:172.21.15.155:3300/0 conn(0x7f43b00b00f0 0x7f43b013bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:43.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.611+0000 7f43b560a640 1 -- 172.21.15.194:0/2482927076 >> 172.21.15.194:0/2482927076 conn(0x7f43b00b16b0 msgr2=0x7f43b006b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:43.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.611+0000 7f43b560a640 1 -- 172.21.15.194:0/2482927076 shutdown_connections 2024-08-09T14:49:43.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.611+0000 7f43b560a640 1 -- 172.21.15.194:0/2482927076 wait complete. 2024-08-09T14:49:43.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.611+0000 7f43b560a640 10 monclient: build_initial_monmap 2024-08-09T14:49:43.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.611+0000 7f43b560a640 10 monclient: monmap: 2024-08-09T14:49:43.619 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:43.619 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:49:43.619 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-08-09T14:49:43.619 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-08-09T14:49:43.619 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:43.619 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:43.619 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:49:43.619 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:49:43.619 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:49:43.619 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:43.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.612+0000 7f43b560a640 1 librados: starting msgr at 2024-08-09T14:49:43.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.612+0000 7f43b560a640 1 librados: starting objecter 2024-08-09T14:49:43.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.612+0000 7f43b560a640 1 Processor -- start 2024-08-09T14:49:43.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.612+0000 7f43b560a640 1 -- start start 2024-08-09T14:49:43.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.612+0000 7f43b560a640 1 librados: setting wanted keys 2024-08-09T14:49:43.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.612+0000 7f43b560a640 1 librados: calling monclient init 2024-08-09T14:49:43.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.612+0000 7f43b560a640 10 monclient: init 2024-08-09T14:49:43.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:43.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 10 monclient: _add_conns ranks=[2,0,1] 2024-08-09T14:49:43.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43b00b00f0 0x7f43b0142770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:43.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 10 monclient(hunting): picked mon.noname-c con 0x7f43b00b00f0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:43.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43b013c450 0x7f43b01d3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:43.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 10 monclient(hunting): picked mon.noname-a con 0x7f43b013c450 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:43.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43b013f4c0 0x7f43b01d41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:43.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 10 monclient(hunting): picked mon.noname-b con 0x7f43b013f4c0 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:49:43.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:43.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f43b00ae540 con 0x7f43b013c450 2024-08-09T14:49:43.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:43.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f43b013cac0 con 0x7f43b013f4c0 2024-08-09T14:49:43.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:43.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f43b00aea60 con 0x7f43b00b00f0 2024-08-09T14:49:43.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:43.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43b560a640 10 monclient(hunting): authenticate will time out at 1208.647221s 2024-08-09T14:49:43.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43af7fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43b013f4c0 0x7f43b01d41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:43.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43af7fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43b013f4c0 0x7f43b01d41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:46288/0 (socket says 172.21.15.194:46288) 2024-08-09T14:49:43.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43af7fe640 1 -- 172.21.15.194:0/4254027210 learned_addr learned my addr 172.21.15.194:0/4254027210 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-08-09T14:49:43.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43af7fe640 10 monclient(hunting): get_auth_request con 0x7f43b013f4c0 auth_method 0 2024-08-09T14:49:43.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:43.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43af7fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:43.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.613+0000 7f43af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:43.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.614+0000 7f43af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:43.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.614+0000 7f43af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:43.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.614+0000 7f43af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:43.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.617+0000 7f43af7fe640 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2024-08-09T14:49:43.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.617+0000 7f43af7fe640 1 -- 172.21.15.194:0/4254027210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43b00b00f0 msgr2=0x7f43b0142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:43.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.617+0000 7f43af7fe640 1 --2- 172.21.15.194:0/4254027210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43b00b00f0 0x7f43b0142770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:43.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.617+0000 7f43af7fe640 1 -- 172.21.15.194:0/4254027210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43b013c450 msgr2=0x7f43b01d3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:43.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.617+0000 7f43af7fe640 1 --2- 172.21.15.194:0/4254027210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43b013c450 0x7f43b01d3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:43.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.617+0000 7f43af7fe640 10 monclient: _finish_hunting 0 2024-08-09T14:49:43.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.617+0000 7f43af7fe640 1 monclient: found mon.noname-b 2024-08-09T14:49:43.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.617+0000 7f43af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:43.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.617+0000 7f43af7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:49:43.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.617+0000 7f43af7fe640 1 -- 172.21.15.194:0/4254027210 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b0141b10 con 0x7f43b013f4c0 2024-08-09T14:49:43.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.619+0000 7f43af7fe640 1 --2- 172.21.15.194:0/4254027210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43b013f4c0 0x7f43b01d41a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f43a800b970 tx=0x7f43a800be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:43.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.619+0000 7f43977fe640 10 client.?.objecter ms_handle_connect 0x7f43b013f4c0 2024-08-09T14:49:43.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.619+0000 7f43977fe640 10 client.?.objecter resend_mon_ops 2024-08-09T14:49:43.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43977fe640 1 -- 172.21.15.194:0/4254027210 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a8019070 con 0x7f43b013f4c0 2024-08-09T14:49:43.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:43.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:49:43.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43977fe640 10 monclient: dump: 2024-08-09T14:49:43.626 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:43.626 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:43.626 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:43.626 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:43.626 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:43.627 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:43.627 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:43.627 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:43.627 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:43.627 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:43.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43977fe640 10 monclient: _finish_auth 0 2024-08-09T14:49:43.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43977fe640 10 monclient: _check_auth_tickets 2024-08-09T14:49:43.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:43.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43b560a640 5 monclient: authenticate success, global_id 4108 2024-08-09T14:49:43.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43b560a640 10 monclient: _renew_subs 2024-08-09T14:49:43.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43b560a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:43.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43b560a640 1 -- 172.21.15.194:0/4254027210 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43b006f430 con 0x7f43b013f4c0 2024-08-09T14:49:43.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43b560a640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:49:43.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43b560a640 10 monclient: _renew_subs 2024-08-09T14:49:43.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43b560a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:43.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43b560a640 1 -- 172.21.15.194:0/4254027210 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43b0142cb0 con 0x7f43b013f4c0 2024-08-09T14:49:43.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.621+0000 7f43b560a640 1 librados: init done 2024-08-09T14:49:43.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.622+0000 7f43b560a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:43.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.622+0000 7f43b560a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:43.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.622+0000 7f43b560a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:43.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.622+0000 7f43b560a640 1 -- 172.21.15.194:0/4254027210 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43b00ac860 con 0x7f43b013f4c0 2024-08-09T14:49:43.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.622+0000 7f43977fe640 1 -- 172.21.15.194:0/4254027210 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a800cb40 con 0x7f43b013f4c0 2024-08-09T14:49:43.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.622+0000 7f43977fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:43.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.622+0000 7f43977fe640 1 -- 172.21.15.194:0/4254027210 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a8023060 con 0x7f43b013f4c0 2024-08-09T14:49:43.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.622+0000 7f43977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:43.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.622+0000 7f43977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:49:43.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.622+0000 7f43977fe640 10 monclient: dump: 2024-08-09T14:49:43.630 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:43.630 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:43.630 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:43.630 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:43.630 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:43.630 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:43.630 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:43.630 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:43.630 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:43.630 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:43.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.622+0000 7f43ad7fa640 4 set_mon_vals no callback set 2024-08-09T14:49:43.825 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.824+0000 7f43977fe640 1 -- 172.21.15.194:0/4254027210 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f43a801c070 con 0x7f43b013f4c0 2024-08-09T14:49:43.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.824+0000 7f43977fe640 1 -- 172.21.15.194:0/4254027210 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f43a80149f0 con 0x7f43b013f4c0 2024-08-09T14:49:43.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.824+0000 7f43977fe640 10 client.4108.objecter ms_dispatch 0x7f43b0000b90 osd_map(2..2 src has 1..2) 2024-08-09T14:49:43.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.824+0000 7f43977fe640 3 client.4108.objecter handle_osd_map got epochs [2,2] > 0 2024-08-09T14:49:43.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.824+0000 7f43977fe640 3 client.4108.objecter handle_osd_map decoding full epoch 2 2024-08-09T14:49:43.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.824+0000 7f43977fe640 20 client.4108.objecter dump_active .. 0 homeless 2024-08-09T14:49:43.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.824+0000 7f43977fe640 1 -- 172.21.15.194:0/4254027210 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7f43b006f430 con 0x7f43b013f4c0 2024-08-09T14:49:43.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.824+0000 7f43977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:43.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.824+0000 7f43977fe640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:49:43.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.952+0000 7f43b560a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6526d474-661b-48e6-bcc7-6288c7477f78", "id": 0}] 2024-08-09T14:49:43.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.952+0000 7f43b560a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6526d474-661b-48e6-bcc7-6288c7477f78", "id": 0}] 2024-08-09T14:49:43.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.952+0000 7f43b560a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:43.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:43.952+0000 7f43b560a640 1 -- 172.21.15.194:0/4254027210 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6526d474-661b-48e6-bcc7-6288c7477f78", "id": 0} v 0) -- 0x7f43b0068ac0 con 0x7f43b013f4c0 2024-08-09T14:49:44.038 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:44.036+0000 7f2c8b84e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-09T14:49:44.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.140+0000 7f43977fe640 1 -- 172.21.15.194:0/4254027210 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6526d474-661b-48e6-bcc7-6288c7477f78", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f43a801e580 con 0x7f43b013f4c0 2024-08-09T14:49:44.141 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.140+0000 7f43977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6526d474-661b-48e6-bcc7-6288c7477f78", "id": 0}] 2024-08-09T14:49:44.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.140+0000 7f43977fe640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:49:44.142 INFO:teuthology.orchestra.run.smithi194.stdout:0 2024-08-09T14:49:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 10 librados: watch_flush enter 2024-08-09T14:49:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 10 librados: watch_flush exit 2024-08-09T14:49:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 20 client.4108.objecter shutdown clearing up homeless session... 2024-08-09T14:49:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 10 client.4108.objecter successfully canceled tick 2024-08-09T14:49:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 10 monclient: shutdown 2024-08-09T14:49:44.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:44.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 1 -- 172.21.15.194:0/4254027210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43b013f4c0 msgr2=0x7f43b01d41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:44.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 1 --2- 172.21.15.194:0/4254027210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43b013f4c0 0x7f43b01d41a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f43a800b970 tx=0x7f43a800be30 comp rx=0 tx=0).stop 2024-08-09T14:49:44.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 1 -- 172.21.15.194:0/4254027210 shutdown_connections 2024-08-09T14:49:44.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 1 --2- 172.21.15.194:0/4254027210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43b013f4c0 0x7f43b01d41a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 1 --2- 172.21.15.194:0/4254027210 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43b013c450 0x7f43b01d3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 1 --2- 172.21.15.194:0/4254027210 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43b00b00f0 0x7f43b0142770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 1 -- 172.21.15.194:0/4254027210 >> 172.21.15.194:0/4254027210 conn(0x7f43b00b16b0 msgr2=0x7f43b006c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:44.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 1 -- 172.21.15.194:0/4254027210 shutdown_connections 2024-08-09T14:49:44.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 1 -- 172.21.15.194:0/4254027210 wait complete. 2024-08-09T14:49:44.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.141+0000 7f43b560a640 1 librados: shutdown 2024-08-09T14:49:44.146 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:44.145+0000 7f2c8b84e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-09T14:49:44.154 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 87673c5a-b124-419e-98da-17a7b7fbdf25 1 2024-08-09T14:49:44.194 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:44.193+0000 7f33aaa46040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-08-09T14:49:44.209 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:44.208+0000 7f2c8b84e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-09T14:49:44.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.252+0000 7f127409e640 10 monclient: get_monmap_and_config 2024-08-09T14:49:44.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.252+0000 7f127409e640 10 monclient: build_initial_monmap 2024-08-09T14:49:44.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.252+0000 7f127409e640 10 monclient: monmap: 2024-08-09T14:49:44.254 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:44.254 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:44.254 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:49:44.253642+0000 2024-08-09T14:49:44.254 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:49:44.253642+0000 2024-08-09T14:49:44.254 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:44.254 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:44.254 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:49:44.254 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:49:44.255 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:49:44.255 INFO:teuthology.orchestra.run.smithi194.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:49:44.255 INFO:teuthology.orchestra.run.smithi194.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:49:44.255 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:44.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.253+0000 7f127409e640 1 Processor -- start 2024-08-09T14:49:44.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.253+0000 7f127409e640 1 -- start start 2024-08-09T14:49:44.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.253+0000 7f127409e640 10 monclient: init 2024-08-09T14:49:44.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f127409e640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:44.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f127409e640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-08-09T14:49:44.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f127409e640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f126c06d470 0x7f126c146150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:44.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f127409e640 10 monclient(hunting): picked mon.noname-b con 0x7f126c06d470 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:44.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f127409e640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f126c0b44a0 0x7f126c13ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:44.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f127409e640 10 monclient(hunting): picked mon.noname-a con 0x7f126c0b44a0 addr v2:172.21.15.155:3300/0 2024-08-09T14:49:44.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f127409e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f126c0b4e50 addr v1:172.21.15.155:6789/0 2024-08-09T14:49:44.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f127409e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f126c0b4930 con 0x7f126c0b4e50 2024-08-09T14:49:44.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f127409e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:44.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f127409e640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f126c06d850 con 0x7f126c0b44a0 2024-08-09T14:49:44.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f127409e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:44.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f127409e640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f126c0b2e10 con 0x7f126c06d470 2024-08-09T14:49:44.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f127409e640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:44.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f127409e640 10 monclient(hunting): authenticate will time out at 1209.288165s 2024-08-09T14:49:44.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f1271639640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f126c0b44a0 0x7f126c13ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:44.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.254+0000 7f1271e3a640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f126c06d470 0x7f126c146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:44.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.255+0000 7f1271639640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f126c0b44a0 0x7f126c13ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.194:53942/0 (socket says 172.21.15.194:53942) 2024-08-09T14:49:44.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.255+0000 7f1271e3a640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f126c06d470 0x7f126c146150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.194:35762/0 (socket says 172.21.15.194:35762) 2024-08-09T14:49:44.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.255+0000 7f1271639640 1 -- 172.21.15.194:0/2013955420 learned_addr learned my addr 172.21.15.194:0/2013955420 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-08-09T14:49:44.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.255+0000 7f1271639640 10 monclient(hunting): get_auth_request con 0x7f126c0b44a0 auth_method 0 2024-08-09T14:49:44.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.255+0000 7f1271639640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:44.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.255+0000 7f1271639640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:44.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.255+0000 7f1271639640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:44.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.255+0000 7f1271e3a640 10 monclient(hunting): get_auth_request con 0x7f126c06d470 auth_method 0 2024-08-09T14:49:44.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.255+0000 7f1271e3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:44.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.255+0000 7f1271e3a640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:44.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.255+0000 7f1271e3a640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:44.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.255+0000 7f1271e3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:44.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.255+0000 7f1271e3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:44.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.255+0000 7f1271e3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:44.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.256+0000 7f1271639640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:44.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.256+0000 7f1271639640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:44.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.256+0000 7f1271639640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:44.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.257+0000 7f1271e3a640 10 monclient(hunting): handle_auth_done global_id 4106 payload 274 2024-08-09T14:49:44.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.257+0000 7f1271e3a640 1 -- 172.21.15.194:0/2013955420 >> v2:172.21.15.155:3300/0 conn(0x7f126c0b44a0 msgr2=0x7f126c13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:44.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.257+0000 7f1271e3a640 1 --2- 172.21.15.194:0/2013955420 >> v2:172.21.15.155:3300/0 conn(0x7f126c0b44a0 0x7f126c13ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.257+0000 7f1271e3a640 1 -- 172.21.15.194:0/2013955420 >> v1:172.21.15.155:6789/0 conn(0x7f126c0b4e50 legacy=0x7f126c0b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:44.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.257+0000 7f1271e3a640 10 monclient: _finish_hunting 0 2024-08-09T14:49:44.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.257+0000 7f1271e3a640 1 monclient: found mon.noname-b 2024-08-09T14:49:44.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.257+0000 7f1271e3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:44.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.257+0000 7f1271e3a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:49:44.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.257+0000 7f1271e3a640 1 -- 172.21.15.194:0/2013955420 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f126c146690 con 0x7f126c06d470 2024-08-09T14:49:44.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.259+0000 7f1271e3a640 1 -- 172.21.15.194:0/2013955420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f126c06d470 msgr2=0x7f126c146150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-08-09T14:49:44.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.259+0000 7f1271e3a640 1 -- 172.21.15.194:0/2013955420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f126c06d470 msgr2=0x7f126c146150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-08-09T14:49:44.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.259+0000 7f1271e3a640 1 --2- 172.21.15.194:0/2013955420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f126c06d470 0x7f126c146150 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f12680097d0 tx=0x7f126802b810 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-08-09T14:49:44.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.259+0000 7f1271e3a640 1 --2- 172.21.15.194:0/2013955420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f126c06d470 0x7f126c146150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T14:49:44.293 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:44.292+0000 7f33aaa46040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-08-09T14:49:44.334 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:44.333+0000 7f33aaa46040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-08-09T14:49:44.377 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:44.376+0000 7f2c8b84e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-09T14:49:44.424 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:44.423+0000 7f2c8b84e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-09T14:49:44.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.460+0000 7f1271e3a640 1 --2- 172.21.15.194:0/2013955420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f126c06d470 0x7f126c146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:44.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1271e3a640 10 monclient: get_auth_request con 0x7f126c06d470 auth_method 0 2024-08-09T14:49:44.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1271e3a640 0 --2- 172.21.15.194:0/2013955420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f126c06d470 0x7f126c146150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-09T14:49:44.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1271e3a640 1 --2- 172.21.15.194:0/2013955420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f126c06d470 0x7f126c146150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1270e38640 10 monclient: ms_handle_reset current mon [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:44.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1270e38640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:44.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1270e38640 1 -- 172.21.15.194:0/2013955420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f126c06d470 msgr2=0x7f126c146150 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-08-09T14:49:44.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1270e38640 1 --2- 172.21.15.194:0/2013955420 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f126c06d470 0x7f126c146150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1270e38640 10 monclient: _add_conns ranks=[4,1] 2024-08-09T14:49:44.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1270e38640 1 --2- 172.21.15.194:0/2013955420 >> v2:172.21.15.194:3300/0 conn(0x7f1258003500 0x7f1258005980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:44.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1270e38640 10 monclient(hunting): picked mon.noname-c con 0x7f1258003500 addr v2:172.21.15.194:3300/0 2024-08-09T14:49:44.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1270e38640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12580062b0 addr v1:172.21.15.194:6789/0 2024-08-09T14:49:44.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1270e38640 1 -- 172.21.15.194:0/2013955420 --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f125800a260 con 0x7f12580062b0 2024-08-09T14:49:44.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1270e38640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:44.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1270e38640 1 -- 172.21.15.194:0/2013955420 --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f125800b4a0 con 0x7f1258003500 2024-08-09T14:49:44.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1270e38640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:44.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1270e38640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:44.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.461+0000 7f1271e3a640 1 --2- 172.21.15.194:0/2013955420 >> v2:172.21.15.194:3300/0 conn(0x7f1258003500 0x7f1258005980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:44.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.462+0000 7f1271e3a640 10 monclient(hunting): get_auth_request con 0x7f1258003500 auth_method 0 2024-08-09T14:49:44.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.462+0000 7f1271e3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:44.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.462+0000 7f1271e3a640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:44.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.462+0000 7f1271e3a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:49:44.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.462+0000 7f1271e3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:44.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.462+0000 7f1271e3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:44.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.462+0000 7f1271e3a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-08-09T14:49:44.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.464+0000 7f1271e3a640 10 monclient(hunting): handle_auth_done global_id 4106 payload 293 2024-08-09T14:49:44.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.464+0000 7f1271e3a640 1 -- 172.21.15.194:0/2013955420 >> v1:172.21.15.194:6789/0 conn(0x7f12580062b0 legacy=0x7f1258008770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:44.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.464+0000 7f1271e3a640 10 monclient: _finish_hunting 0 2024-08-09T14:49:44.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.464+0000 7f1271e3a640 1 monclient: found mon.noname-c 2024-08-09T14:49:44.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.464+0000 7f1271e3a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:44.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.464+0000 7f1271e3a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:49:44.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.464+0000 7f1271e3a640 1 -- 172.21.15.194:0/2013955420 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125800b710 con 0x7f1258003500 2024-08-09T14:49:44.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.466+0000 7f1271e3a640 1 --2- 172.21.15.194:0/2013955420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1258003500 0x7f1258005980 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f126c0b2f90 tx=0x7f1268009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11ab7cd05c71198c server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:44.474 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f1270e38640 1 -- 172.21.15.194:0/2013955420 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126802f790 con 0x7f1258003500 2024-08-09T14:49:44.474 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f1270e38640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:44.474 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f1270e38640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:49:44.474 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f1270e38640 10 monclient: dump: 2024-08-09T14:49:44.475 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:44.475 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:44.475 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.475 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.475 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:44.475 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:44.475 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:44.475 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:44.475 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:44.475 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:44.475 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f1270e38640 10 monclient: _finish_auth 0 2024-08-09T14:49:44.475 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f1270e38640 10 monclient: _check_auth_tickets 2024-08-09T14:49:44.475 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f1270e38640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:44.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f1270e38640 1 -- 172.21.15.194:0/2013955420 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f126802fcb0 con 0x7f1258003500 2024-08-09T14:49:44.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f1270e38640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:44.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f1270e38640 1 -- 172.21.15.194:0/2013955420 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f126802fe60 con 0x7f1258003500 2024-08-09T14:49:44.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f127409e640 5 monclient: authenticate success, global_id 4106 2024-08-09T14:49:44.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f127409e640 10 monclient: get_monmap_and_config success 2024-08-09T14:49:44.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f127409e640 4 set_mon_vals no callback set 2024-08-09T14:49:44.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f1270e38640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:44.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f1270e38640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:49:44.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.473+0000 7f1270e38640 10 monclient: dump: 2024-08-09T14:49:44.476 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:44.476 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:44.477 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.477 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.477 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:44.477 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:44.477 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:44.477 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:44.477 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:44.477 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:44.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.474+0000 7f127409e640 10 monclient: shutdown 2024-08-09T14:49:44.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.474+0000 7f127409e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:44.480 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.474+0000 7f127409e640 1 -- 172.21.15.194:0/2013955420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1258003500 msgr2=0x7f1258005980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:44.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.474+0000 7f127409e640 1 --2- 172.21.15.194:0/2013955420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1258003500 0x7f1258005980 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f126c0b2f90 tx=0x7f1268009190 comp rx=0 tx=0).stop 2024-08-09T14:49:44.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.474+0000 7f127263b640 1 -- 172.21.15.194:0/2013955420 reap_dead start 2024-08-09T14:49:44.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.474+0000 7f127409e640 1 -- 172.21.15.194:0/2013955420 shutdown_connections 2024-08-09T14:49:44.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.474+0000 7f127409e640 1 --2- 172.21.15.194:0/2013955420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1258003500 0x7f1258005980 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.474+0000 7f127409e640 1 -- 172.21.15.194:0/2013955420 >> 172.21.15.194:0/2013955420 conn(0x7f126c0b8810 msgr2=0x7f126c0b8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:44.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.474+0000 7f127409e640 1 -- 172.21.15.194:0/2013955420 shutdown_connections 2024-08-09T14:49:44.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.474+0000 7f127409e640 1 -- 172.21.15.194:0/2013955420 wait complete. 2024-08-09T14:49:44.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.475+0000 7f127409e640 10 monclient: build_initial_monmap 2024-08-09T14:49:44.481 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.475+0000 7f127409e640 10 monclient: monmap: 2024-08-09T14:49:44.481 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:44.481 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:49:44.481 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-08-09T14:49:44.481 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-08-09T14:49:44.482 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:44.482 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:44.482 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:49:44.482 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:49:44.482 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:49:44.482 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:44.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.475+0000 7f127409e640 1 librados: starting msgr at 2024-08-09T14:49:44.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.475+0000 7f127409e640 1 librados: starting objecter 2024-08-09T14:49:44.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.475+0000 7f127409e640 1 Processor -- start 2024-08-09T14:49:44.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.475+0000 7f127409e640 1 -- start start 2024-08-09T14:49:44.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.475+0000 7f127409e640 1 librados: setting wanted keys 2024-08-09T14:49:44.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.475+0000 7f127409e640 1 librados: calling monclient init 2024-08-09T14:49:44.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.475+0000 7f127409e640 10 monclient: init 2024-08-09T14:49:44.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:44.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 10 monclient: _add_conns ranks=[2,1,0] 2024-08-09T14:49:44.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1258003500 0x7f126c0ad600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:44.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 10 monclient(hunting): picked mon.noname-c con 0x7f1258003500 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:44.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c0adb40 0x7f126c1dbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:44.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 10 monclient(hunting): picked mon.noname-b con 0x7f126c0adb40 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:49:44.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f126c1dc230 0x7f126c13c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:44.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 10 monclient(hunting): picked mon.noname-a con 0x7f126c1dc230 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:44.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:44.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f126c0aad80 con 0x7f126c1dc230 2024-08-09T14:49:44.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:44.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f126c0aac00 con 0x7f126c0adb40 2024-08-09T14:49:44.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:44.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f126c140450 con 0x7f1258003500 2024-08-09T14:49:44.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:44.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127409e640 10 monclient(hunting): authenticate will time out at 1209.510289s 2024-08-09T14:49:44.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f1271639640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c0adb40 0x7f126c1dbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:44.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f1271e3a640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1258003500 0x7f126c0ad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:44.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.476+0000 7f127263b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f126c1dc230 0x7f126c13c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:44.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271639640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c0adb40 0x7f126c1dbcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:46316/0 (socket says 172.21.15.194:46316) 2024-08-09T14:49:44.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271639640 1 -- 172.21.15.194:0/1511559893 learned_addr learned my addr 172.21.15.194:0/1511559893 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-08-09T14:49:44.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271639640 10 monclient(hunting): get_auth_request con 0x7f126c0adb40 auth_method 0 2024-08-09T14:49:44.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271639640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:44.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271639640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:44.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271639640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:44.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271e3a640 10 monclient(hunting): get_auth_request con 0x7f1258003500 auth_method 0 2024-08-09T14:49:44.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271e3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:44.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271e3a640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:44.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271e3a640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:44.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f127263b640 10 monclient(hunting): get_auth_request con 0x7f126c1dc230 auth_method 0 2024-08-09T14:49:44.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f127263b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:44.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f127263b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:44.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f127263b640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:44.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271639640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:44.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271639640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:44.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271639640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:44.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271e3a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:44.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271e3a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:44.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f1271e3a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:44.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f127263b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:44.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.477+0000 7f127263b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:44.486 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.478+0000 7f127263b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:44.487 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:44.477+0000 7f2c8b84e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-09T14:49:44.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.480+0000 7f1271639640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-08-09T14:49:44.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.480+0000 7f1271639640 1 -- 172.21.15.194:0/1511559893 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1258003500 msgr2=0x7f126c0ad600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:44.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.480+0000 7f1271639640 1 --2- 172.21.15.194:0/1511559893 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1258003500 0x7f126c0ad600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.480+0000 7f1271639640 1 -- 172.21.15.194:0/1511559893 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f126c1dc230 msgr2=0x7f126c13c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:44.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.480+0000 7f1271639640 1 --2- 172.21.15.194:0/1511559893 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f126c1dc230 0x7f126c13c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.480+0000 7f1271639640 10 monclient: _finish_hunting 0 2024-08-09T14:49:44.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.480+0000 7f1271639640 1 monclient: found mon.noname-b 2024-08-09T14:49:44.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.480+0000 7f1271639640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:44.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.480+0000 7f1271639640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:49:44.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.480+0000 7f1271639640 1 -- 172.21.15.194:0/1511559893 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f126c1dc6e0 con 0x7f126c0adb40 2024-08-09T14:49:44.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.482+0000 7f1271639640 1 --2- 172.21.15.194:0/1511559893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c0adb40 0x7f126c1dbcf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f125c00e870 tx=0x7f125c00ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:44.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.482+0000 7f12627fc640 10 client.?.objecter ms_handle_connect 0x7f126c0adb40 2024-08-09T14:49:44.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.482+0000 7f12627fc640 10 client.?.objecter resend_mon_ops 2024-08-09T14:49:44.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.486+0000 7f12627fc640 1 -- 172.21.15.194:0/1511559893 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c017070 con 0x7f126c0adb40 2024-08-09T14:49:44.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.486+0000 7f12627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:44.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.486+0000 7f12627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:49:44.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.486+0000 7f12627fc640 10 monclient: dump: 2024-08-09T14:49:44.489 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:44.490 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:44.490 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.490 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.490 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:44.490 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:44.490 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:44.490 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:44.490 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:44.490 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:44.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f12627fc640 10 monclient: _finish_auth 0 2024-08-09T14:49:44.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f12627fc640 10 monclient: _check_auth_tickets 2024-08-09T14:49:44.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f12627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:44.490 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f12627fc640 1 -- 172.21.15.194:0/1511559893 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f125c00c940 con 0x7f126c0adb40 2024-08-09T14:49:44.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f12627fc640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:44.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f12627fc640 1 -- 172.21.15.194:0/1511559893 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f125c00cd10 con 0x7f126c0adb40 2024-08-09T14:49:44.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f127409e640 5 monclient: authenticate success, global_id 4111 2024-08-09T14:49:44.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f127409e640 10 monclient: _renew_subs 2024-08-09T14:49:44.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f127409e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:44.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f127409e640 1 -- 172.21.15.194:0/1511559893 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f126c0aed70 con 0x7f126c0adb40 2024-08-09T14:49:44.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f12627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:44.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f12627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:49:44.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f12627fc640 10 monclient: dump: 2024-08-09T14:49:44.491 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:44.491 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:44.491 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.491 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:44.492 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f1263fff640 4 set_mon_vals no callback set 2024-08-09T14:49:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f127409e640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:49:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f127409e640 10 monclient: _renew_subs 2024-08-09T14:49:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f127409e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f127409e640 1 -- 172.21.15.194:0/1511559893 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f126c141ba0 con 0x7f126c0adb40 2024-08-09T14:49:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f127409e640 1 librados: init done 2024-08-09T14:49:44.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.487+0000 7f127409e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:44.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.488+0000 7f127409e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:44.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.488+0000 7f127409e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:44.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.488+0000 7f127409e640 1 -- 172.21.15.194:0/1511559893 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f126c146150 con 0x7f126c0adb40 2024-08-09T14:49:44.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.490+0000 7f12627fc640 1 -- 172.21.15.194:0/1511559893 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f125c01a070 con 0x7f126c0adb40 2024-08-09T14:49:44.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.490+0000 7f12627fc640 1 -- 172.21.15.194:0/1511559893 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f125c006420 con 0x7f126c0adb40 2024-08-09T14:49:44.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.490+0000 7f12627fc640 10 client.4111.objecter ms_dispatch 0x7f126c000b90 osd_map(3..3 src has 1..3) 2024-08-09T14:49:44.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.490+0000 7f12627fc640 3 client.4111.objecter handle_osd_map got epochs [3,3] > 0 2024-08-09T14:49:44.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.490+0000 7f12627fc640 3 client.4111.objecter handle_osd_map decoding full epoch 3 2024-08-09T14:49:44.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.490+0000 7f12627fc640 20 client.4111.objecter dump_active .. 0 homeless 2024-08-09T14:49:44.504 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:44.503+0000 7f33aaa46040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-08-09T14:49:44.562 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:44.560+0000 7f33aaa46040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-08-09T14:49:44.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.585+0000 7f12627fc640 1 -- 172.21.15.194:0/1511559893 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7f125c006bf0 con 0x7f126c0adb40 2024-08-09T14:49:44.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.585+0000 7f12627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:44.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.585+0000 7f12627fc640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:49:44.592 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:44.590+0000 7f2c8b84e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-09T14:49:44.615 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:44.614+0000 7f33aaa46040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-08-09T14:49:44.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.656+0000 7f127409e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "87673c5a-b124-419e-98da-17a7b7fbdf25", "id": 1}] 2024-08-09T14:49:44.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.656+0000 7f127409e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "87673c5a-b124-419e-98da-17a7b7fbdf25", "id": 1}] 2024-08-09T14:49:44.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.656+0000 7f127409e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:44.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.656+0000 7f127409e640 1 -- 172.21.15.194:0/1511559893 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd new", "uuid": "87673c5a-b124-419e-98da-17a7b7fbdf25", "id": 1} v 0) -- 0x7f126c068960 con 0x7f126c0adb40 2024-08-09T14:49:44.732 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:44.730+0000 7f33aaa46040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-08-09T14:49:44.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.750+0000 7f12627fc640 1 -- 172.21.15.194:0/1511559893 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "87673c5a-b124-419e-98da-17a7b7fbdf25", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f125c0066f0 con 0x7f126c0adb40 2024-08-09T14:49:44.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.750+0000 7f12627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "87673c5a-b124-419e-98da-17a7b7fbdf25", "id": 1}] 2024-08-09T14:49:44.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.750+0000 7f12627fc640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:49:44.752 INFO:teuthology.orchestra.run.smithi194.stdout:1 2024-08-09T14:49:44.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 10 librados: watch_flush enter 2024-08-09T14:49:44.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 10 librados: watch_flush exit 2024-08-09T14:49:44.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 20 client.4111.objecter shutdown clearing up homeless session... 2024-08-09T14:49:44.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 10 client.4111.objecter successfully canceled tick 2024-08-09T14:49:44.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 10 monclient: shutdown 2024-08-09T14:49:44.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:44.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 1 -- 172.21.15.194:0/1511559893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c0adb40 msgr2=0x7f126c1dbcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:44.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 1 --2- 172.21.15.194:0/1511559893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c0adb40 0x7f126c1dbcf0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f125c00e870 tx=0x7f125c00ec70 comp rx=0 tx=0).stop 2024-08-09T14:49:44.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 1 -- 172.21.15.194:0/1511559893 shutdown_connections 2024-08-09T14:49:44.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 1 --2- 172.21.15.194:0/1511559893 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f126c1dc230 0x7f126c13c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 1 --2- 172.21.15.194:0/1511559893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f126c0adb40 0x7f126c1dbcf0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 1 --2- 172.21.15.194:0/1511559893 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1258003500 0x7f126c0ad600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 1 -- 172.21.15.194:0/1511559893 >> 172.21.15.194:0/1511559893 conn(0x7f126c0b8810 msgr2=0x7f126c0b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:44.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 1 -- 172.21.15.194:0/1511559893 shutdown_connections 2024-08-09T14:49:44.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 1 -- 172.21.15.194:0/1511559893 wait complete. 2024-08-09T14:49:44.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.751+0000 7f127409e640 1 librados: shutdown 2024-08-09T14:49:44.763 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new a14a708a-903f-4de2-8228-79e79ed135e6 2 2024-08-09T14:49:44.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.899+0000 7f795c331640 10 monclient: get_monmap_and_config 2024-08-09T14:49:44.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.899+0000 7f795c331640 10 monclient: build_initial_monmap 2024-08-09T14:49:44.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.899+0000 7f795c331640 10 monclient: monmap: 2024-08-09T14:49:44.901 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:44.901 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:44.901 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:49:44.900308+0000 2024-08-09T14:49:44.901 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:49:44.900308+0000 2024-08-09T14:49:44.901 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:44.901 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:44.901 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:49:44.901 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:49:44.901 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:49:44.901 INFO:teuthology.orchestra.run.smithi194.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:49:44.901 INFO:teuthology.orchestra.run.smithi194.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:49:44.902 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:44.902 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.899+0000 7f795c331640 1 Processor -- start 2024-08-09T14:49:44.902 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 1 -- start start 2024-08-09T14:49:44.902 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 10 monclient: init 2024-08-09T14:49:44.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:44.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-08-09T14:49:44.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f795406d470 0x7f7954146150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:44.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 10 monclient(hunting): picked mon.noname-c con 0x7f795406d470 addr v2:172.21.15.194:3300/0 2024-08-09T14:49:44.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79540b44a0 0x7f795413ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:44.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 10 monclient(hunting): picked mon.noname-b con 0x7f79540b44a0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:44.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f79540b4e50 addr v1:172.21.15.194:6789/0 2024-08-09T14:49:44.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79540b4930 con 0x7f79540b4e50 2024-08-09T14:49:44.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:44.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f795406d850 con 0x7f795406d470 2024-08-09T14:49:44.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:44.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f79540b2e10 con 0x7f79540b44a0 2024-08-09T14:49:44.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:44.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.900+0000 7f795c331640 10 monclient(hunting): authenticate will time out at 1209.934813s 2024-08-09T14:49:44.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.901+0000 7f795a0cd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f795406d470 0x7f7954146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:44.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.901+0000 7f79598cc640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79540b44a0 0x7f795413ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:44.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.901+0000 7f795a0cd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f795406d470 0x7f7954146150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:46326/0 (socket says 172.21.15.194:46326) 2024-08-09T14:49:44.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.901+0000 7f795a0cd640 1 -- 172.21.15.194:0/2771018987 learned_addr learned my addr 172.21.15.194:0/2771018987 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-08-09T14:49:44.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.901+0000 7f795a0cd640 10 monclient(hunting): get_auth_request con 0x7f795406d470 auth_method 0 2024-08-09T14:49:44.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.901+0000 7f795a0cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:44.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.901+0000 7f795a0cd640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:44.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.901+0000 7f795a0cd640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:44.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.901+0000 7f79598cc640 10 monclient(hunting): get_auth_request con 0x7f79540b44a0 auth_method 0 2024-08-09T14:49:44.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.901+0000 7f79598cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:44.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.901+0000 7f79598cc640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:44.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.901+0000 7f79598cc640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:44.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.902+0000 7f79598cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:44.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.902+0000 7f79598cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:44.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.902+0000 7f79598cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:44.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.902+0000 7f795a0cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:44.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.902+0000 7f795a0cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:44.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.902+0000 7f795a0cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:44.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.904+0000 7f79598cc640 10 monclient(hunting): handle_auth_done global_id 4112 payload 274 2024-08-09T14:49:44.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.904+0000 7f79598cc640 1 -- 172.21.15.194:0/2771018987 >> v2:172.21.15.194:3300/0 conn(0x7f795406d470 msgr2=0x7f7954146150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:44.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.904+0000 7f79598cc640 1 --2- 172.21.15.194:0/2771018987 >> v2:172.21.15.194:3300/0 conn(0x7f795406d470 0x7f7954146150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.904+0000 7f79598cc640 1 -- 172.21.15.194:0/2771018987 >> v1:172.21.15.194:6789/0 conn(0x7f79540b4e50 legacy=0x7f79540b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:44.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.904+0000 7f79598cc640 10 monclient: _finish_hunting 0 2024-08-09T14:49:44.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.904+0000 7f79598cc640 1 monclient: found mon.noname-b 2024-08-09T14:49:44.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.904+0000 7f79598cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:44.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.904+0000 7f79598cc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:49:44.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.904+0000 7f79598cc640 1 -- 172.21.15.194:0/2771018987 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7954146690 con 0x7f79540b44a0 2024-08-09T14:49:44.908 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.906+0000 7f79598cc640 1 --2- 172.21.15.194:0/2771018987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79540b44a0 0x7f795413ff10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f79440097d0 tx=0x7f794402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=221080494a6b1657 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:44.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f79590cb640 1 -- 172.21.15.194:0/2771018987 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7944031770 con 0x7f79540b44a0 2024-08-09T14:49:44.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f79590cb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:44.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f79590cb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:49:44.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f79590cb640 10 monclient: dump: 2024-08-09T14:49:44.911 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:44.911 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:44.911 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.911 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.911 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:44.911 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:44.911 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:44.911 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:44.912 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:44.912 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:44.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f79590cb640 10 monclient: _finish_auth 0 2024-08-09T14:49:44.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f79590cb640 10 monclient: _check_auth_tickets 2024-08-09T14:49:44.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f79590cb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:44.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f79590cb640 1 -- 172.21.15.194:0/2771018987 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7944037070 con 0x7f79540b44a0 2024-08-09T14:49:44.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f79590cb640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:44.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f79590cb640 1 -- 172.21.15.194:0/2771018987 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7944031b30 con 0x7f79540b44a0 2024-08-09T14:49:44.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f795c331640 5 monclient: authenticate success, global_id 4112 2024-08-09T14:49:44.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f795c331640 10 monclient: get_monmap_and_config success 2024-08-09T14:49:44.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f79590cb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:44.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f795c331640 4 set_mon_vals no callback set 2024-08-09T14:49:44.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f79590cb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:49:44.913 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.909+0000 7f79590cb640 10 monclient: dump: 2024-08-09T14:49:44.913 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:44.913 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:44.913 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.913 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.913 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:44.913 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:44.913 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:44.913 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:44.913 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:44.913 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:44.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.910+0000 7f795c331640 10 monclient: shutdown 2024-08-09T14:49:44.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.910+0000 7f795c331640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:44.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.910+0000 7f795c331640 1 -- 172.21.15.194:0/2771018987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79540b44a0 msgr2=0x7f795413ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:44.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.910+0000 7f795c331640 1 --2- 172.21.15.194:0/2771018987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79540b44a0 0x7f795413ff10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f79440097d0 tx=0x7f794402b810 comp rx=0 tx=0).stop 2024-08-09T14:49:44.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.910+0000 7f795c331640 1 -- 172.21.15.194:0/2771018987 shutdown_connections 2024-08-09T14:49:44.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.910+0000 7f795c331640 1 --2- 172.21.15.194:0/2771018987 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79540b44a0 0x7f795413ff10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.910+0000 7f795c331640 1 --2- 172.21.15.194:0/2771018987 >> v2:172.21.15.194:3300/0 conn(0x7f795406d470 0x7f7954146150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.910+0000 7f795c331640 1 -- 172.21.15.194:0/2771018987 >> 172.21.15.194:0/2771018987 conn(0x7f79540b8810 msgr2=0x7f79540b8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:44.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.910+0000 7f795c331640 1 -- 172.21.15.194:0/2771018987 shutdown_connections 2024-08-09T14:49:44.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.910+0000 7f795c331640 1 -- 172.21.15.194:0/2771018987 wait complete. 2024-08-09T14:49:44.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.911+0000 7f795c331640 10 monclient: build_initial_monmap 2024-08-09T14:49:44.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.911+0000 7f795c331640 10 monclient: monmap: 2024-08-09T14:49:44.917 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:44.917 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:49:44.918 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-08-09T14:49:44.918 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-08-09T14:49:44.918 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:44.918 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:44.918 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:49:44.918 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:49:44.918 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:49:44.918 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:44.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.911+0000 7f795c331640 1 librados: starting msgr at 2024-08-09T14:49:44.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.911+0000 7f795c331640 1 librados: starting objecter 2024-08-09T14:49:44.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.911+0000 7f795c331640 1 Processor -- start 2024-08-09T14:49:44.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.911+0000 7f795c331640 1 -- start start 2024-08-09T14:49:44.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.911+0000 7f795c331640 1 librados: setting wanted keys 2024-08-09T14:49:44.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.911+0000 7f795c331640 1 librados: calling monclient init 2024-08-09T14:49:44.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.911+0000 7f795c331640 10 monclient: init 2024-08-09T14:49:44.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:44.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 10 monclient: _add_conns ranks=[0,2,1] 2024-08-09T14:49:44.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f795406d470 0x7f7954141c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:44.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 10 monclient(hunting): picked mon.noname-a con 0x7f795406d470 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:44.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79540b4e50 0x7f79541ccb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:44.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 10 monclient(hunting): picked mon.noname-c con 0x7f79540b4e50 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:44.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79541cd040 0x7f79541db9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:44.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 10 monclient(hunting): picked mon.noname-b con 0x7f79541cd040 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:49:44.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:44.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f79540aad80 con 0x7f795406d470 2024-08-09T14:49:44.919 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:44.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f79540aac00 con 0x7f79541cd040 2024-08-09T14:49:44.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:44.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f7954140450 con 0x7f79540b4e50 2024-08-09T14:49:44.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:44.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795c331640 10 monclient(hunting): authenticate will time out at 1209.946337s 2024-08-09T14:49:44.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795a8ce640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79541cd040 0x7f79541db9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:44.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f79598cc640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79540b4e50 0x7f79541ccb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:44.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.912+0000 7f795a0cd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f795406d470 0x7f7954141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:44.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f79598cc640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79540b4e50 0x7f79541ccb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.194:35792/0 (socket says 172.21.15.194:35792) 2024-08-09T14:49:44.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f79598cc640 1 -- 172.21.15.194:0/887824989 learned_addr learned my addr 172.21.15.194:0/887824989 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-08-09T14:49:44.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f79598cc640 10 monclient(hunting): get_auth_request con 0x7f79540b4e50 auth_method 0 2024-08-09T14:49:44.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f79598cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:44.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f79598cc640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:44.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f79598cc640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:44.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f795a0cd640 10 monclient(hunting): get_auth_request con 0x7f795406d470 auth_method 0 2024-08-09T14:49:44.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f795a0cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:44.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f795a0cd640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:44.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f795a0cd640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:44.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f795a8ce640 10 monclient(hunting): get_auth_request con 0x7f79541cd040 auth_method 0 2024-08-09T14:49:44.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f795a8ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:44.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f795a8ce640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:44.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f795a8ce640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:44.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f79598cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:44.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f79598cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:44.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f79598cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:44.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f795a0cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:44.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f795a0cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:44.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.913+0000 7f795a0cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:44.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.914+0000 7f795a8ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:44.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.914+0000 7f795a8ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:44.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.914+0000 7f795a8ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:44.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.916+0000 7f795a0cd640 10 monclient(hunting): handle_auth_done global_id 4122 payload 824 2024-08-09T14:49:44.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.916+0000 7f795a0cd640 1 -- 172.21.15.194:0/887824989 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79540b4e50 msgr2=0x7f79541ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:44.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.916+0000 7f795a0cd640 1 --2- 172.21.15.194:0/887824989 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79540b4e50 0x7f79541ccb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.916+0000 7f795a0cd640 1 -- 172.21.15.194:0/887824989 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79541cd040 msgr2=0x7f79541db9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:44.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.916+0000 7f795a0cd640 1 --2- 172.21.15.194:0/887824989 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79541cd040 0x7f79541db9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:44.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.916+0000 7f795a0cd640 10 monclient: _finish_hunting 0 2024-08-09T14:49:44.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.916+0000 7f795a0cd640 1 monclient: found mon.noname-a 2024-08-09T14:49:44.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.916+0000 7f795a0cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:44.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.916+0000 7f795a0cd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:49:44.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.916+0000 7f795a0cd640 1 -- 172.21.15.194:0/887824989 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7954142d50 con 0x7f795406d470 2024-08-09T14:49:44.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.916+0000 7f79598cc640 1 --2- 172.21.15.194:0/887824989 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79540b4e50 0x7f79541ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-09T14:49:44.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.919+0000 7f795a0cd640 1 --2- 172.21.15.194:0/887824989 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f795406d470 0x7f7954141c50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f795000e850 tx=0x7f795000ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:44.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.919+0000 7f794affd640 10 client.?.objecter ms_handle_connect 0x7f795406d470 2024-08-09T14:49:44.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.919+0000 7f794affd640 10 client.?.objecter resend_mon_ops 2024-08-09T14:49:44.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.923+0000 7f794affd640 1 -- 172.21.15.194:0/887824989 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950017070 con 0x7f795406d470 2024-08-09T14:49:44.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.923+0000 7f794affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:44.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.923+0000 7f794affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:49:44.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.923+0000 7f794affd640 10 monclient: dump: 2024-08-09T14:49:44.925 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:44.926 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:44.926 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.926 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.926 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:44.926 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:44.926 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:44.926 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:44.926 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:44.926 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:44.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.923+0000 7f794affd640 10 monclient: _finish_auth 0 2024-08-09T14:49:44.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.923+0000 7f794affd640 10 monclient: _check_auth_tickets 2024-08-09T14:49:44.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.923+0000 7f794affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:44.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.923+0000 7f794affd640 1 -- 172.21.15.194:0/887824989 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f795000c940 con 0x7f795406d470 2024-08-09T14:49:44.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f795c331640 5 monclient: authenticate success, global_id 4122 2024-08-09T14:49:44.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f795c331640 10 monclient: _renew_subs 2024-08-09T14:49:44.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f795c331640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:49:44.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f795c331640 1 -- 172.21.15.194:0/887824989 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7954142fc0 con 0x7f795406d470 2024-08-09T14:49:44.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f794affd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:44.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f794affd640 1 -- 172.21.15.194:0/887824989 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f795000ccf0 con 0x7f795406d470 2024-08-09T14:49:44.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f794affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:44.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f794affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:49:44.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f794affd640 10 monclient: dump: 2024-08-09T14:49:44.927 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:44.927 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:44.927 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.927 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:44.928 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:44.928 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:44.928 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:44.928 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:44.928 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:44.928 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:44.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f79588ca640 4 set_mon_vals no callback set 2024-08-09T14:49:44.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f795c331640 10 client.4122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:49:44.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f795c331640 10 monclient: _renew_subs 2024-08-09T14:49:44.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f795c331640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:49:44.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f795c331640 1 -- 172.21.15.194:0/887824989 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7954143580 con 0x7f795406d470 2024-08-09T14:49:44.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f795c331640 1 librados: init done 2024-08-09T14:49:44.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f795c331640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:44.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f795c331640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:44.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f795c331640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:49:44.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.924+0000 7f795c331640 1 -- 172.21.15.194:0/887824989 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7954146150 con 0x7f795406d470 2024-08-09T14:49:44.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.929+0000 7f794affd640 1 -- 172.21.15.194:0/887824989 <== mon.0 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f795001a070 con 0x7f795406d470 2024-08-09T14:49:44.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.929+0000 7f794affd640 1 -- 172.21.15.194:0/887824989 <== mon.0 v2:172.21.15.155:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f7950013470 con 0x7f795406d470 2024-08-09T14:49:44.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.929+0000 7f794affd640 10 client.4122.objecter ms_dispatch 0x7f7954000b90 osd_map(4..4 src has 1..4) 2024-08-09T14:49:44.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.929+0000 7f794affd640 3 client.4122.objecter handle_osd_map got epochs [4,4] > 0 2024-08-09T14:49:44.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.929+0000 7f794affd640 3 client.4122.objecter handle_osd_map decoding full epoch 4 2024-08-09T14:49:44.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:44.929+0000 7f794affd640 20 client.4122.objecter dump_active .. 0 homeless 2024-08-09T14:49:44.985 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:44.984+0000 7f2c8b84e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-09T14:49:45.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.041+0000 7f794affd640 1 -- 172.21.15.194:0/887824989 <== mon.0 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7f7950006490 con 0x7f795406d470 2024-08-09T14:49:45.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.041+0000 7f794affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:45.043 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.041+0000 7f794affd640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:49:45.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.112+0000 7f795c331640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a14a708a-903f-4de2-8228-79e79ed135e6", "id": 2}] 2024-08-09T14:49:45.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.112+0000 7f795c331640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a14a708a-903f-4de2-8228-79e79ed135e6", "id": 2}] 2024-08-09T14:49:45.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.112+0000 7f795c331640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:49:45.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.112+0000 7f795c331640 1 -- 172.21.15.194:0/887824989 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a14a708a-903f-4de2-8228-79e79ed135e6", "id": 2} v 0) -- 0x7f795413eb90 con 0x7f795406d470 2024-08-09T14:49:45.125 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:45.124+0000 7f33aaa46040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-08-09T14:49:45.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.147+0000 7f794affd640 1 -- 172.21.15.194:0/887824989 <== mon.0 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a14a708a-903f-4de2-8228-79e79ed135e6", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f79500067a0 con 0x7f795406d470 2024-08-09T14:49:45.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.147+0000 7f794affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a14a708a-903f-4de2-8228-79e79ed135e6", "id": 2}] 2024-08-09T14:49:45.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.147+0000 7f794affd640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:49:45.148 INFO:teuthology.orchestra.run.smithi194.stdout:2 2024-08-09T14:49:45.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 10 librados: watch_flush enter 2024-08-09T14:49:45.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 10 librados: watch_flush exit 2024-08-09T14:49:45.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 20 client.4122.objecter shutdown clearing up homeless session... 2024-08-09T14:49:45.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 10 client.4122.objecter successfully canceled tick 2024-08-09T14:49:45.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 10 monclient: shutdown 2024-08-09T14:49:45.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:45.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 1 -- 172.21.15.194:0/887824989 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f795406d470 msgr2=0x7f7954141c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:45.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 1 --2- 172.21.15.194:0/887824989 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f795406d470 0x7f7954141c50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f795000e850 tx=0x7f795000ed10 comp rx=0 tx=0).stop 2024-08-09T14:49:45.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 1 -- 172.21.15.194:0/887824989 shutdown_connections 2024-08-09T14:49:45.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 1 --2- 172.21.15.194:0/887824989 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79541cd040 0x7f79541db9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 1 --2- 172.21.15.194:0/887824989 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f79540b4e50 0x7f79541ccb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 1 --2- 172.21.15.194:0/887824989 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f795406d470 0x7f7954141c50 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 1 -- 172.21.15.194:0/887824989 >> 172.21.15.194:0/887824989 conn(0x7f79540b8810 msgr2=0x7f79540b5a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:45.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 1 -- 172.21.15.194:0/887824989 shutdown_connections 2024-08-09T14:49:45.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 1 -- 172.21.15.194:0/887824989 wait complete. 2024-08-09T14:49:45.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.148+0000 7f795c331640 1 librados: shutdown 2024-08-09T14:49:45.160 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 8e139d50-e25c-4428-9edf-c696189cb40e 3 2024-08-09T14:49:45.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.258+0000 7f68ea987640 10 monclient: get_monmap_and_config 2024-08-09T14:49:45.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.258+0000 7f68ea987640 10 monclient: build_initial_monmap 2024-08-09T14:49:45.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.258+0000 7f68ea987640 10 monclient: monmap: 2024-08-09T14:49:45.260 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:45.260 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:45.260 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:49:45.259636+0000 2024-08-09T14:49:45.260 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:49:45.259636+0000 2024-08-09T14:49:45.260 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:45.260 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:45.260 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:49:45.260 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:49:45.260 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:49:45.260 INFO:teuthology.orchestra.run.smithi194.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:49:45.261 INFO:teuthology.orchestra.run.smithi194.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:49:45.261 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:45.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.259+0000 7f68ea987640 1 Processor -- start 2024-08-09T14:49:45.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.259+0000 7f68ea987640 1 -- start start 2024-08-09T14:49:45.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.259+0000 7f68ea987640 10 monclient: init 2024-08-09T14:49:45.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68ea987640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:45.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68ea987640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-08-09T14:49:45.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68ea987640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68e406d470 0x7f68e4146150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:45.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68ea987640 10 monclient(hunting): picked mon.noname-b con 0x7f68e406d470 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:45.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68ea987640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68e40b44a0 addr v1:172.21.15.155:6789/0 2024-08-09T14:49:45.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68ea987640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f68e4140620 0x7f68e414d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:45.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68ea987640 10 monclient(hunting): picked mon.noname-c con 0x7f68e4140620 addr v2:172.21.15.194:3300/0 2024-08-09T14:49:45.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68ea987640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68e40b4930 con 0x7f68e40b44a0 2024-08-09T14:49:45.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68ea987640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:45.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68ea987640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f68e406d850 con 0x7f68e4140620 2024-08-09T14:49:45.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68ea987640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:45.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68ea987640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f68e40b2e10 con 0x7f68e406d470 2024-08-09T14:49:45.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68ea987640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:45.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68ea987640 10 monclient(hunting): authenticate will time out at 1210.294282s 2024-08-09T14:49:45.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.260+0000 7f68e8f24640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f68e4140620 0x7f68e414d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:45.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.261+0000 7f68e8f24640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f68e4140620 0x7f68e414d480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:46352/0 (socket says 172.21.15.194:46352) 2024-08-09T14:49:45.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.261+0000 7f68e8f24640 1 -- 172.21.15.194:0/670462177 learned_addr learned my addr 172.21.15.194:0/670462177 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-08-09T14:49:45.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.261+0000 7f68e8f24640 10 monclient(hunting): get_auth_request con 0x7f68e4140620 auth_method 0 2024-08-09T14:49:45.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.261+0000 7f68e8f24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:45.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.261+0000 7f68e8f24640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:45.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.261+0000 7f68e8f24640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:45.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.261+0000 7f68e3fff640 1 --2- 172.21.15.194:0/670462177 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68e406d470 0x7f68e4146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:45.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.262+0000 7f68e3fff640 10 monclient(hunting): get_auth_request con 0x7f68e406d470 auth_method 0 2024-08-09T14:49:45.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.262+0000 7f68e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:45.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.262+0000 7f68e3fff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:45.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.262+0000 7f68e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:45.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.262+0000 7f68e8f24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:45.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.262+0000 7f68e8f24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:45.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.262+0000 7f68e8f24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:45.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.263+0000 7f68e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:45.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.263+0000 7f68e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:45.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.263+0000 7f68e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:45.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.264+0000 7f68e8f24640 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2024-08-09T14:49:45.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.264+0000 7f68e8f24640 1 -- 172.21.15.194:0/670462177 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68e406d470 msgr2=0x7f68e4146150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:45.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.264+0000 7f68e8f24640 1 --2- 172.21.15.194:0/670462177 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68e406d470 0x7f68e4146150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.264+0000 7f68e8f24640 1 -- 172.21.15.194:0/670462177 >> v1:172.21.15.155:6789/0 conn(0x7f68e40b44a0 legacy=0x7f68e413ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:45.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.264+0000 7f68e8f24640 10 monclient: _finish_hunting 0 2024-08-09T14:49:45.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.264+0000 7f68e8f24640 1 monclient: found mon.noname-c 2024-08-09T14:49:45.267 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.264+0000 7f68e8f24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:45.267 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.264+0000 7f68e8f24640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:49:45.267 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.264+0000 7f68e8f24640 1 -- 172.21.15.194:0/670462177 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e40b4e50 con 0x7f68e4140620 2024-08-09T14:49:45.267 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.264+0000 7f68e3fff640 1 --2- 172.21.15.194:0/670462177 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68e406d470 0x7f68e4146150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-09T14:49:45.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.267+0000 7f68e8f24640 1 --2- 172.21.15.194:0/670462177 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68e4140620 0x7f68e414d480 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f68dc009e50 tx=0x7f68dc02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1e636417a529b3a server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:45.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68e2ffd640 1 -- 172.21.15.194:0/670462177 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68dc02cb00 con 0x7f68e4140620 2024-08-09T14:49:45.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:45.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:49:45.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68e2ffd640 10 monclient: dump: 2024-08-09T14:49:45.274 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:45.274 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:45.274 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.274 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.274 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:45.274 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:45.274 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:45.274 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:45.274 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:45.274 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:45.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68e2ffd640 10 monclient: _finish_auth 0 2024-08-09T14:49:45.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68e2ffd640 10 monclient: _check_auth_tickets 2024-08-09T14:49:45.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:45.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68e2ffd640 1 -- 172.21.15.194:0/670462177 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68dc02cca0 con 0x7f68e4140620 2024-08-09T14:49:45.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68e2ffd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:45.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68ea987640 5 monclient: authenticate success, global_id 4120 2024-08-09T14:49:45.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68e2ffd640 1 -- 172.21.15.194:0/670462177 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68dc034610 con 0x7f68e4140620 2024-08-09T14:49:45.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:45.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:49:45.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68e2ffd640 10 monclient: dump: 2024-08-09T14:49:45.275 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:45.275 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:45.276 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.276 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.276 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:45.276 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:45.276 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:45.276 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:45.276 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:45.276 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:45.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68ea987640 10 monclient: get_monmap_and_config success 2024-08-09T14:49:45.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.272+0000 7f68ea987640 4 set_mon_vals no callback set 2024-08-09T14:49:45.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.273+0000 7f68ea987640 10 monclient: shutdown 2024-08-09T14:49:45.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.273+0000 7f68ea987640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:45.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.273+0000 7f68ea987640 1 -- 172.21.15.194:0/670462177 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68e4140620 msgr2=0x7f68e414d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:45.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.273+0000 7f68ea987640 1 --2- 172.21.15.194:0/670462177 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68e4140620 0x7f68e414d480 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f68dc009e50 tx=0x7f68dc02bf60 comp rx=0 tx=0).stop 2024-08-09T14:49:45.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.273+0000 7f68ea987640 1 -- 172.21.15.194:0/670462177 shutdown_connections 2024-08-09T14:49:45.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.273+0000 7f68ea987640 1 --2- 172.21.15.194:0/670462177 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68e4140620 0x7f68e414d480 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.273+0000 7f68ea987640 1 --2- 172.21.15.194:0/670462177 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68e406d470 0x7f68e4146150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.273+0000 7f68ea987640 1 -- 172.21.15.194:0/670462177 >> 172.21.15.194:0/670462177 conn(0x7f68e40b8810 msgr2=0x7f68e40b8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:45.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.273+0000 7f68ea987640 1 -- 172.21.15.194:0/670462177 shutdown_connections 2024-08-09T14:49:45.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.273+0000 7f68ea987640 1 -- 172.21.15.194:0/670462177 wait complete. 2024-08-09T14:49:45.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.274+0000 7f68ea987640 10 monclient: build_initial_monmap 2024-08-09T14:49:45.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.274+0000 7f68ea987640 10 monclient: monmap: 2024-08-09T14:49:45.280 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:45.280 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:49:45.280 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-08-09T14:49:45.280 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-08-09T14:49:45.281 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:45.281 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:45.281 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:49:45.281 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:49:45.281 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:49:45.281 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:45.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.274+0000 7f68ea987640 1 librados: starting msgr at 2024-08-09T14:49:45.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.274+0000 7f68ea987640 1 librados: starting objecter 2024-08-09T14:49:45.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.274+0000 7f68ea987640 1 Processor -- start 2024-08-09T14:49:45.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.274+0000 7f68ea987640 1 -- start start 2024-08-09T14:49:45.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.274+0000 7f68ea987640 1 librados: setting wanted keys 2024-08-09T14:49:45.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.274+0000 7f68ea987640 1 librados: calling monclient init 2024-08-09T14:49:45.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.274+0000 7f68ea987640 10 monclient: init 2024-08-09T14:49:45.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:45.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 10 monclient: _add_conns ranks=[0,1,2] 2024-08-09T14:49:45.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68e406d470 0x7f68e40b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:45.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 10 monclient(hunting): picked mon.noname-a con 0x7f68e406d470 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:45.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68e4140620 0x7f68e4143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:45.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 10 monclient(hunting): picked mon.noname-b con 0x7f68e4140620 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:49:45.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68e41dbd20 0x7f68e41378b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:45.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 10 monclient(hunting): picked mon.noname-c con 0x7f68e41dbd20 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:45.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:45.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f68e40aa9f0 con 0x7f68e406d470 2024-08-09T14:49:45.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:45.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f68e41468a0 con 0x7f68e4140620 2024-08-09T14:49:45.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f68e40aab70 con 0x7f68e41dbd20 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68ea987640 10 monclient(hunting): authenticate will time out at 1210.309429s 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.275+0000 7f68e37fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68e4140620 0x7f68e4143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e8f24640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68e41dbd20 0x7f68e41378b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e3fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68e406d470 0x7f68e40b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e3fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68e406d470 0x7f68e40b7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.194:53964/0 (socket says 172.21.15.194:53964) 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e3fff640 1 -- 172.21.15.194:0/3207571311 learned_addr learned my addr 172.21.15.194:0/3207571311 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e3fff640 10 monclient(hunting): get_auth_request con 0x7f68e406d470 auth_method 0 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e3fff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e37fe640 10 monclient(hunting): get_auth_request con 0x7f68e4140620 auth_method 0 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:45.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e37fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:45.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:45.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e8f24640 10 monclient(hunting): get_auth_request con 0x7f68e41dbd20 auth_method 0 2024-08-09T14:49:45.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e8f24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:45.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e8f24640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:45.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e8f24640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:45.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:45.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:45.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.276+0000 7f68e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:45.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.277+0000 7f68e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:45.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.277+0000 7f68e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:45.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.277+0000 7f68e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:45.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.277+0000 7f68e8f24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:45.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.277+0000 7f68e8f24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:45.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.277+0000 7f68e8f24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:45.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.279+0000 7f68e3fff640 10 monclient(hunting): handle_auth_done global_id 4125 payload 824 2024-08-09T14:49:45.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.279+0000 7f68e3fff640 1 -- 172.21.15.194:0/3207571311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68e41dbd20 msgr2=0x7f68e41378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:45.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.279+0000 7f68e3fff640 1 --2- 172.21.15.194:0/3207571311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68e41dbd20 0x7f68e41378b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.279+0000 7f68e3fff640 1 -- 172.21.15.194:0/3207571311 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68e4140620 msgr2=0x7f68e4143310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:45.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.279+0000 7f68e3fff640 1 --2- 172.21.15.194:0/3207571311 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68e4140620 0x7f68e4143310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.279+0000 7f68e3fff640 10 monclient: _finish_hunting 0 2024-08-09T14:49:45.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.279+0000 7f68e3fff640 1 monclient: found mon.noname-a 2024-08-09T14:49:45.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.279+0000 7f68e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:45.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.279+0000 7f68e3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:49:45.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.279+0000 7f68e3fff640 1 -- 172.21.15.194:0/3207571311 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68e41ccd90 con 0x7f68e406d470 2024-08-09T14:49:45.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.282+0000 7f68e3fff640 1 --2- 172.21.15.194:0/3207571311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68e406d470 0x7f68e40b7db0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f68d400ec10 tx=0x7f68d400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:45.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.282+0000 7f68e0ff9640 10 client.?.objecter ms_handle_connect 0x7f68e406d470 2024-08-09T14:49:45.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.282+0000 7f68e0ff9640 10 client.?.objecter resend_mon_ops 2024-08-09T14:49:45.287 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.285+0000 7f68e0ff9640 1 -- 172.21.15.194:0/3207571311 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4017070 con 0x7f68e406d470 2024-08-09T14:49:45.287 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.285+0000 7f68e0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:45.287 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.285+0000 7f68e0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:49:45.287 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.285+0000 7f68e0ff9640 10 monclient: dump: 2024-08-09T14:49:45.288 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:45.288 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:45.288 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.288 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.288 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:45.288 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:45.288 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:45.288 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:45.288 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:45.288 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:45.288 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.285+0000 7f68e0ff9640 10 monclient: _finish_auth 0 2024-08-09T14:49:45.288 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.285+0000 7f68e0ff9640 10 monclient: _check_auth_tickets 2024-08-09T14:49:45.288 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.285+0000 7f68e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:45.288 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.285+0000 7f68e0ff9640 1 -- 172.21.15.194:0/3207571311 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68d400cc30 con 0x7f68e406d470 2024-08-09T14:49:45.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.285+0000 7f68e0ff9640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:45.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.285+0000 7f68e0ff9640 1 -- 172.21.15.194:0/3207571311 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68d4013470 con 0x7f68e406d470 2024-08-09T14:49:45.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.285+0000 7f68ea987640 5 monclient: authenticate success, global_id 4125 2024-08-09T14:49:45.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.285+0000 7f68ea987640 10 monclient: _renew_subs 2024-08-09T14:49:45.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.286+0000 7f68ea987640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:49:45.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.286+0000 7f68ea987640 1 -- 172.21.15.194:0/3207571311 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68e41cd2e0 con 0x7f68e406d470 2024-08-09T14:49:45.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.285+0000 7f68e27fc640 4 set_mon_vals no callback set 2024-08-09T14:49:45.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.286+0000 7f68e0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:45.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.286+0000 7f68e0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:49:45.289 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.286+0000 7f68e0ff9640 10 monclient: dump: 2024-08-09T14:49:45.289 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:45.289 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:45.289 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.290 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.290 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:45.290 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:45.290 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:45.290 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:45.290 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:45.290 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:45.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.286+0000 7f68ea987640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:49:45.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.286+0000 7f68ea987640 10 monclient: _renew_subs 2024-08-09T14:49:45.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.286+0000 7f68ea987640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:49:45.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.286+0000 7f68ea987640 1 -- 172.21.15.194:0/3207571311 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68e41cd000 con 0x7f68e406d470 2024-08-09T14:49:45.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.286+0000 7f68ea987640 1 librados: init done 2024-08-09T14:49:45.291 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.287+0000 7f68ea987640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:45.291 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.287+0000 7f68ea987640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:45.291 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.287+0000 7f68ea987640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:49:45.291 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.287+0000 7f68ea987640 1 -- 172.21.15.194:0/3207571311 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68e4146150 con 0x7f68e406d470 2024-08-09T14:49:45.293 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:45.292+0000 7f2c8b84e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-09T14:49:45.347 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:45.346+0000 7f2c8b84e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-09T14:49:45.382 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.381+0000 7f68e0ff9640 1 -- 172.21.15.194:0/3207571311 <== mon.0 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f68d401a070 con 0x7f68e406d470 2024-08-09T14:49:45.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.385+0000 7f68e0ff9640 1 -- 172.21.15.194:0/3207571311 <== mon.0 v2:172.21.15.155:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f68d4005fd0 con 0x7f68e406d470 2024-08-09T14:49:45.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.385+0000 7f68e0ff9640 10 client.4125.objecter ms_dispatch 0x7f68e4000b90 osd_map(5..5 src has 1..5) 2024-08-09T14:49:45.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.385+0000 7f68e0ff9640 3 client.4125.objecter handle_osd_map got epochs [5,5] > 0 2024-08-09T14:49:45.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.385+0000 7f68e0ff9640 3 client.4125.objecter handle_osd_map decoding full epoch 5 2024-08-09T14:49:45.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.385+0000 7f68e0ff9640 20 client.4125.objecter dump_active .. 0 homeless 2024-08-09T14:49:45.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.386+0000 7f68e0ff9640 1 -- 172.21.15.194:0/3207571311 <== mon.0 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7f68e41cd2e0 con 0x7f68e406d470 2024-08-09T14:49:45.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.386+0000 7f68e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:45.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.386+0000 7f68e0ff9640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:49:45.406 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:45.405+0000 7f33aaa46040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-08-09T14:49:45.459 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.458+0000 7f68ea987640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8e139d50-e25c-4428-9edf-c696189cb40e", "id": 3}] 2024-08-09T14:49:45.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.458+0000 7f68ea987640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8e139d50-e25c-4428-9edf-c696189cb40e", "id": 3}] 2024-08-09T14:49:45.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.458+0000 7f68ea987640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:49:45.460 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.458+0000 7f68ea987640 1 -- 172.21.15.194:0/3207571311 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8e139d50-e25c-4428-9edf-c696189cb40e", "id": 3} v 0) -- 0x7f68e40b9040 con 0x7f68e406d470 2024-08-09T14:49:45.460 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:45.459+0000 7f33aaa46040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-08-09T14:49:45.465 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:45.464+0000 7f2c8b84e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-09T14:49:45.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.506+0000 7f68e0ff9640 1 -- 172.21.15.194:0/3207571311 <== mon.0 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8e139d50-e25c-4428-9edf-c696189cb40e", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f68d4005990 con 0x7f68e406d470 2024-08-09T14:49:45.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.506+0000 7f68e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8e139d50-e25c-4428-9edf-c696189cb40e", "id": 3}] 2024-08-09T14:49:45.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.506+0000 7f68e0ff9640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:49:45.508 INFO:teuthology.orchestra.run.smithi194.stdout:3 2024-08-09T14:49:45.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.507+0000 7f68ea987640 10 librados: watch_flush enter 2024-08-09T14:49:45.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.507+0000 7f68ea987640 10 librados: watch_flush exit 2024-08-09T14:49:45.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.507+0000 7f68ea987640 20 client.4125.objecter shutdown clearing up homeless session... 2024-08-09T14:49:45.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.507+0000 7f68ea987640 10 client.4125.objecter successfully canceled tick 2024-08-09T14:49:45.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.507+0000 7f68ea987640 10 monclient: shutdown 2024-08-09T14:49:45.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.507+0000 7f68ea987640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:45.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.507+0000 7f68ea987640 1 -- 172.21.15.194:0/3207571311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68e406d470 msgr2=0x7f68e40b7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:45.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.507+0000 7f68ea987640 1 --2- 172.21.15.194:0/3207571311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68e406d470 0x7f68e40b7db0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f68d400ec10 tx=0x7f68d400c520 comp rx=0 tx=0).stop 2024-08-09T14:49:45.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.508+0000 7f68ea987640 1 -- 172.21.15.194:0/3207571311 shutdown_connections 2024-08-09T14:49:45.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.508+0000 7f68ea987640 1 --2- 172.21.15.194:0/3207571311 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68e41dbd20 0x7f68e41378b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.508+0000 7f68ea987640 1 --2- 172.21.15.194:0/3207571311 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68e4140620 0x7f68e4143310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.508+0000 7f68ea987640 1 --2- 172.21.15.194:0/3207571311 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68e406d470 0x7f68e40b7db0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.508+0000 7f68ea987640 1 -- 172.21.15.194:0/3207571311 >> 172.21.15.194:0/3207571311 conn(0x7f68e40b8810 msgr2=0x7f68e40b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.508+0000 7f68ea987640 1 -- 172.21.15.194:0/3207571311 shutdown_connections 2024-08-09T14:49:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.508+0000 7f68ea987640 1 -- 172.21.15.194:0/3207571311 wait complete. 2024-08-09T14:49:45.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.508+0000 7f68ea987640 1 librados: shutdown 2024-08-09T14:49:45.519 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new d0331386-2f50-4bf9-b72e-24c80f9fe9a8 4 2024-08-09T14:49:45.583 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:45.582+0000 7f33aaa46040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-08-09T14:49:45.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.617+0000 7f6432df9640 10 monclient: get_monmap_and_config 2024-08-09T14:49:45.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.617+0000 7f6432df9640 10 monclient: build_initial_monmap 2024-08-09T14:49:45.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.617+0000 7f6432df9640 10 monclient: monmap: 2024-08-09T14:49:45.618 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:45.618 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:45.618 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:49:45.618094+0000 2024-08-09T14:49:45.618 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:49:45.618094+0000 2024-08-09T14:49:45.618 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:45.619 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:45.619 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:49:45.619 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:49:45.619 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:49:45.619 INFO:teuthology.orchestra.run.smithi194.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:49:45.619 INFO:teuthology.orchestra.run.smithi194.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:49:45.619 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:45.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.617+0000 7f6432df9640 1 Processor -- start 2024-08-09T14:49:45.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 1 -- start start 2024-08-09T14:49:45.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 10 monclient: init 2024-08-09T14:49:45.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:45.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-08-09T14:49:45.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f642c06d470 0x7f642c146150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:45.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 10 monclient(hunting): picked mon.noname-a con 0x7f642c06d470 addr v2:172.21.15.155:3300/0 2024-08-09T14:49:45.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f642c0b44a0 0x7f642c13ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:45.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 10 monclient(hunting): picked mon.noname-c con 0x7f642c0b44a0 addr v2:172.21.15.194:3300/0 2024-08-09T14:49:45.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f642c0b4e50 0x7f642c140450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:45.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 10 monclient(hunting): picked mon.noname-b con 0x7f642c0b4e50 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:45.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:45.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f642c06d850 con 0x7f642c06d470 2024-08-09T14:49:45.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:45.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f642c0b4930 con 0x7f642c0b44a0 2024-08-09T14:49:45.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:45.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f642c0b2e10 con 0x7f642c0b4e50 2024-08-09T14:49:45.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:45.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.618+0000 7f6432df9640 10 monclient(hunting): authenticate will time out at 1210.652723s 2024-08-09T14:49:45.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.619+0000 7f642bfff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f642c0b44a0 0x7f642c13ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:45.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.619+0000 7f6431396640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f642c0b4e50 0x7f642c140450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:45.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.619+0000 7f6430b95640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f642c06d470 0x7f642c146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:45.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.619+0000 7f642bfff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f642c0b44a0 0x7f642c13ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:46380/0 (socket says 172.21.15.194:46380) 2024-08-09T14:49:45.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.619+0000 7f642bfff640 1 -- 172.21.15.194:0/1214268466 learned_addr learned my addr 172.21.15.194:0/1214268466 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-08-09T14:49:45.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.619+0000 7f642bfff640 10 monclient(hunting): get_auth_request con 0x7f642c0b44a0 auth_method 0 2024-08-09T14:49:45.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.619+0000 7f642bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:45.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.619+0000 7f642bfff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:45.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.619+0000 7f642bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:45.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.619+0000 7f6431396640 10 monclient(hunting): get_auth_request con 0x7f642c0b4e50 auth_method 0 2024-08-09T14:49:45.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.619+0000 7f6431396640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:45.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.619+0000 7f6431396640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:45.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.619+0000 7f6431396640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:45.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.619+0000 7f6430b95640 10 monclient(hunting): get_auth_request con 0x7f642c06d470 auth_method 0 2024-08-09T14:49:45.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.620+0000 7f6430b95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:45.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.620+0000 7f6430b95640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:45.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.620+0000 7f6430b95640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:45.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.620+0000 7f642bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:45.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.620+0000 7f642bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:45.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.620+0000 7f642bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:45.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.620+0000 7f6431396640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:45.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.620+0000 7f6431396640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:45.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.620+0000 7f6431396640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:45.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.620+0000 7f6430b95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:45.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.620+0000 7f6430b95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:45.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.620+0000 7f6430b95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:45.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.621+0000 7f642bfff640 10 monclient(hunting): handle_auth_done global_id 4126 payload 274 2024-08-09T14:49:45.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.621+0000 7f642bfff640 1 -- 172.21.15.194:0/1214268466 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f642c0b4e50 msgr2=0x7f642c140450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:45.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.621+0000 7f642bfff640 1 --2- 172.21.15.194:0/1214268466 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f642c0b4e50 0x7f642c140450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.621+0000 7f642bfff640 1 -- 172.21.15.194:0/1214268466 >> v2:172.21.15.155:3300/0 conn(0x7f642c06d470 msgr2=0x7f642c146150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:45.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.621+0000 7f642bfff640 1 --2- 172.21.15.194:0/1214268466 >> v2:172.21.15.155:3300/0 conn(0x7f642c06d470 0x7f642c146150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.621+0000 7f642bfff640 10 monclient: _finish_hunting 0 2024-08-09T14:49:45.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.621+0000 7f642bfff640 1 monclient: found mon.noname-c 2024-08-09T14:49:45.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.621+0000 7f642bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:45.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.622+0000 7f642bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:49:45.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.622+0000 7f642bfff640 1 -- 172.21.15.194:0/1214268466 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f642c146690 con 0x7f642c0b44a0 2024-08-09T14:49:45.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.623+0000 7f642bfff640 1 --2- 172.21.15.194:0/1214268466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f642c0b44a0 0x7f642c13ff10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f642c143d10 tx=0x7f641802b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50862fd7714673fd server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:45.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f642b7fe640 1 -- 172.21.15.194:0/1214268466 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6418035020 con 0x7f642c0b44a0 2024-08-09T14:49:45.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f642b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:45.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f642b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:49:45.628 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f642b7fe640 10 monclient: dump: 2024-08-09T14:49:45.628 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:45.628 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:45.629 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.629 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.629 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:45.629 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:45.629 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:45.629 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:45.629 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:45.629 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:45.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f642b7fe640 10 monclient: _finish_auth 0 2024-08-09T14:49:45.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f642b7fe640 10 monclient: _check_auth_tickets 2024-08-09T14:49:45.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f642b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:45.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f642b7fe640 1 -- 172.21.15.194:0/1214268466 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6418031870 con 0x7f642c0b44a0 2024-08-09T14:49:45.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f6432df9640 5 monclient: authenticate success, global_id 4126 2024-08-09T14:49:45.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f6432df9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:49:45.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f642b7fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:45.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f642b7fe640 1 -- 172.21.15.194:0/1214268466 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6418031c40 con 0x7f642c0b44a0 2024-08-09T14:49:45.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f6432df9640 10 monclient: get_monmap_and_config success 2024-08-09T14:49:45.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f6432df9640 4 set_mon_vals no callback set 2024-08-09T14:49:45.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f642b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:45.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f642b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:49:45.630 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f642b7fe640 10 monclient: dump: 2024-08-09T14:49:45.630 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:45.630 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:45.630 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.631 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.631 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:45.631 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:45.631 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:45.631 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:45.631 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:45.631 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:45.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f6432df9640 10 monclient: shutdown 2024-08-09T14:49:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f6432df9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f6432df9640 1 -- 172.21.15.194:0/1214268466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f642c0b44a0 msgr2=0x7f642c13ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f6432df9640 1 --2- 172.21.15.194:0/1214268466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f642c0b44a0 0x7f642c13ff10 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f642c143d10 tx=0x7f641802b910 comp rx=0 tx=0).stop 2024-08-09T14:49:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f6432df9640 1 -- 172.21.15.194:0/1214268466 shutdown_connections 2024-08-09T14:49:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f6432df9640 1 --2- 172.21.15.194:0/1214268466 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f642c0b4e50 0x7f642c140450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.627+0000 7f6432df9640 1 --2- 172.21.15.194:0/1214268466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f642c0b44a0 0x7f642c13ff10 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.628+0000 7f6432df9640 1 --2- 172.21.15.194:0/1214268466 >> v2:172.21.15.155:3300/0 conn(0x7f642c06d470 0x7f642c146150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.628+0000 7f6432df9640 1 -- 172.21.15.194:0/1214268466 >> 172.21.15.194:0/1214268466 conn(0x7f642c0b8810 msgr2=0x7f642c0b8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.628+0000 7f6432df9640 1 -- 172.21.15.194:0/1214268466 shutdown_connections 2024-08-09T14:49:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.628+0000 7f6432df9640 1 -- 172.21.15.194:0/1214268466 wait complete. 2024-08-09T14:49:45.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.628+0000 7f6432df9640 10 monclient: build_initial_monmap 2024-08-09T14:49:45.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.628+0000 7f6432df9640 10 monclient: monmap: 2024-08-09T14:49:45.640 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:45.640 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:49:45.640 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-08-09T14:49:45.640 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-08-09T14:49:45.640 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:45.640 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:45.640 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:49:45.640 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:49:45.640 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:49:45.641 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:45.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.628+0000 7f6432df9640 1 librados: starting msgr at 2024-08-09T14:49:45.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.628+0000 7f6432df9640 1 librados: starting objecter 2024-08-09T14:49:45.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 1 Processor -- start 2024-08-09T14:49:45.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 1 -- start start 2024-08-09T14:49:45.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 1 librados: setting wanted keys 2024-08-09T14:49:45.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 1 librados: calling monclient init 2024-08-09T14:49:45.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 10 monclient: init 2024-08-09T14:49:45.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:45.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 10 monclient: _add_conns ranks=[1,0,2] 2024-08-09T14:49:45.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f642c06d470 0x7f642c0b76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:45.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 10 monclient(hunting): picked mon.noname-b con 0x7f642c06d470 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:49:45.641 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f642c0b4e50 0x7f642c1d7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:45.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 10 monclient(hunting): picked mon.noname-a con 0x7f642c0b4e50 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:45.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f642c143270 0x7f642c1d8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:45.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 10 monclient(hunting): picked mon.noname-c con 0x7f642c143270 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:45.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:45.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f642c0aad80 con 0x7f642c0b4e50 2024-08-09T14:49:45.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:45.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f642c0aac00 con 0x7f642c06d470 2024-08-09T14:49:45.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:45.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f642c0b3210 con 0x7f642c143270 2024-08-09T14:49:45.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:45.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.629+0000 7f6432df9640 10 monclient(hunting): authenticate will time out at 1210.663933s 2024-08-09T14:49:45.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.630+0000 7f6430b95640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f642c06d470 0x7f642c0b76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:45.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.630+0000 7f6431396640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f642c143270 0x7f642c1d8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:45.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.630+0000 7f6430b95640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f642c06d470 0x7f642c0b76c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:46386/0 (socket says 172.21.15.194:46386) 2024-08-09T14:49:45.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.630+0000 7f6430b95640 1 -- 172.21.15.194:0/320230436 learned_addr learned my addr 172.21.15.194:0/320230436 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-08-09T14:49:45.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.630+0000 7f6430b95640 10 monclient(hunting): get_auth_request con 0x7f642c06d470 auth_method 0 2024-08-09T14:49:45.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.630+0000 7f6430b95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:45.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.630+0000 7f6430b95640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:45.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.630+0000 7f6430b95640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:45.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.630+0000 7f6431396640 10 monclient(hunting): get_auth_request con 0x7f642c143270 auth_method 0 2024-08-09T14:49:45.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.630+0000 7f6431396640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:45.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.630+0000 7f6431396640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:45.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.630+0000 7f6431396640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:45.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.631+0000 7f6430b95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:45.643 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.631+0000 7f6430b95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:45.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.631+0000 7f6430b95640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:45.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.631+0000 7f6431396640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:45.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.631+0000 7f6431396640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:45.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.631+0000 7f6431396640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:45.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.631+0000 7f642bfff640 1 --2- 172.21.15.194:0/320230436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f642c0b4e50 0x7f642c1d7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:45.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.632+0000 7f642bfff640 10 monclient(hunting): get_auth_request con 0x7f642c0b4e50 auth_method 0 2024-08-09T14:49:45.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.632+0000 7f642bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:45.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.632+0000 7f642bfff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:45.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.632+0000 7f642bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:45.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.633+0000 7f642bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:45.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.633+0000 7f642bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:45.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.633+0000 7f642bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:45.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.634+0000 7f6430b95640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-08-09T14:49:45.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.634+0000 7f6430b95640 1 -- 172.21.15.194:0/320230436 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f642c143270 msgr2=0x7f642c1d8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:45.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.634+0000 7f6430b95640 1 --2- 172.21.15.194:0/320230436 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f642c143270 0x7f642c1d8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.634+0000 7f6430b95640 1 -- 172.21.15.194:0/320230436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f642c0b4e50 msgr2=0x7f642c1d7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:45.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.634+0000 7f6430b95640 1 --2- 172.21.15.194:0/320230436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f642c0b4e50 0x7f642c1d7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.634+0000 7f6430b95640 10 monclient: _finish_hunting 0 2024-08-09T14:49:45.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.634+0000 7f6430b95640 1 monclient: found mon.noname-b 2024-08-09T14:49:45.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.634+0000 7f6430b95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:45.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.634+0000 7f6430b95640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:49:45.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.634+0000 7f6430b95640 1 -- 172.21.15.194:0/320230436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f642c1cc550 con 0x7f642c06d470 2024-08-09T14:49:45.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.636+0000 7f6430b95640 1 --2- 172.21.15.194:0/320230436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f642c06d470 0x7f642c0b76c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f642000e9b0 tx=0x7f642000ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:45.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.636+0000 7f64297fa640 10 client.?.objecter ms_handle_connect 0x7f642c06d470 2024-08-09T14:49:45.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.636+0000 7f64297fa640 10 client.?.objecter resend_mon_ops 2024-08-09T14:49:45.647 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f64297fa640 1 -- 172.21.15.194:0/320230436 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6420017070 con 0x7f642c06d470 2024-08-09T14:49:45.647 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f64297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:45.647 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f64297fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:49:45.647 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f64297fa640 10 monclient: dump: 2024-08-09T14:49:45.648 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:45.648 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:45.648 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.648 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.648 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:45.648 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:45.648 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:45.648 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:45.648 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:45.648 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:45.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f64297fa640 10 monclient: _finish_auth 0 2024-08-09T14:49:45.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f64297fa640 10 monclient: _check_auth_tickets 2024-08-09T14:49:45.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f64297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:45.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f6432df9640 5 monclient: authenticate success, global_id 4129 2024-08-09T14:49:45.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f6432df9640 10 monclient: _renew_subs 2024-08-09T14:49:45.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f6432df9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:45.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f6432df9640 1 -- 172.21.15.194:0/320230436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f642c1cc7c0 con 0x7f642c06d470 2024-08-09T14:49:45.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f6432df9640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:49:45.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f6432df9640 10 monclient: _renew_subs 2024-08-09T14:49:45.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f6432df9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:45.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f6432df9640 1 -- 172.21.15.194:0/320230436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f642c142610 con 0x7f642c06d470 2024-08-09T14:49:45.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.638+0000 7f6432df9640 1 librados: init done 2024-08-09T14:49:45.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.639+0000 7f6432df9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:45.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.639+0000 7f6432df9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:45.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.639+0000 7f6432df9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:45.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.639+0000 7f6432df9640 1 -- 172.21.15.194:0/320230436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f642c146150 con 0x7f642c06d470 2024-08-09T14:49:45.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.640+0000 7f64297fa640 1 -- 172.21.15.194:0/320230436 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f642000c9a0 con 0x7f642c06d470 2024-08-09T14:49:45.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.640+0000 7f64297fa640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:45.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.640+0000 7f64297fa640 1 -- 172.21.15.194:0/320230436 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6420021060 con 0x7f642c06d470 2024-08-09T14:49:45.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.640+0000 7f64297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:45.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.640+0000 7f64297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:49:45.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.640+0000 7f64297fa640 10 monclient: dump: 2024-08-09T14:49:45.650 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:45.650 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:45.650 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.650 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:45.650 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:45.650 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:45.650 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:45.651 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:45.651 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:45.651 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:45.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.640+0000 7f642affd640 4 set_mon_vals no callback set 2024-08-09T14:49:45.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.742+0000 7f64297fa640 1 -- 172.21.15.194:0/320230436 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f642001a070 con 0x7f642c06d470 2024-08-09T14:49:45.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.742+0000 7f64297fa640 1 -- 172.21.15.194:0/320230436 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f6420013470 con 0x7f642c06d470 2024-08-09T14:49:45.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.742+0000 7f64297fa640 10 client.4129.objecter ms_dispatch 0x7f642c000b90 osd_map(6..6 src has 1..6) 2024-08-09T14:49:45.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.742+0000 7f64297fa640 3 client.4129.objecter handle_osd_map got epochs [6,6] > 0 2024-08-09T14:49:45.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.742+0000 7f64297fa640 3 client.4129.objecter handle_osd_map decoding full epoch 6 2024-08-09T14:49:45.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.742+0000 7f64297fa640 20 client.4129.objecter dump_active .. 0 homeless 2024-08-09T14:49:45.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.742+0000 7f64297fa640 1 -- 172.21.15.194:0/320230436 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7f64200136c0 con 0x7f642c06d470 2024-08-09T14:49:45.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.742+0000 7f64297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:45.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.742+0000 7f64297fa640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:49:45.805 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:45.804+0000 7f2c8b84e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-09T14:49:45.817 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.815+0000 7f6432df9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d0331386-2f50-4bf9-b72e-24c80f9fe9a8", "id": 4}] 2024-08-09T14:49:45.817 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.815+0000 7f6432df9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d0331386-2f50-4bf9-b72e-24c80f9fe9a8", "id": 4}] 2024-08-09T14:49:45.817 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.815+0000 7f6432df9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:45.817 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.815+0000 7f6432df9640 1 -- 172.21.15.194:0/320230436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d0331386-2f50-4bf9-b72e-24c80f9fe9a8", "id": 4} v 0) -- 0x7f642c068960 con 0x7f642c06d470 2024-08-09T14:49:45.858 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:45.857+0000 7f2c8b84e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-09T14:49:45.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.878+0000 7f64297fa640 1 -- 172.21.15.194:0/320230436 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d0331386-2f50-4bf9-b72e-24c80f9fe9a8", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f6420005ff0 con 0x7f642c06d470 2024-08-09T14:49:45.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.878+0000 7f64297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d0331386-2f50-4bf9-b72e-24c80f9fe9a8", "id": 4}] 2024-08-09T14:49:45.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.878+0000 7f64297fa640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:49:45.880 INFO:teuthology.orchestra.run.smithi194.stdout:4 2024-08-09T14:49:45.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.879+0000 7f6432df9640 10 librados: watch_flush enter 2024-08-09T14:49:45.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.879+0000 7f6432df9640 10 librados: watch_flush exit 2024-08-09T14:49:45.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.879+0000 7f6432df9640 20 client.4129.objecter shutdown clearing up homeless session... 2024-08-09T14:49:45.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.879+0000 7f6432df9640 10 client.4129.objecter successfully canceled tick 2024-08-09T14:49:45.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.879+0000 7f6432df9640 10 monclient: shutdown 2024-08-09T14:49:45.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.879+0000 7f6432df9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:45.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.879+0000 7f6432df9640 1 -- 172.21.15.194:0/320230436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f642c06d470 msgr2=0x7f642c0b76c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:45.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.879+0000 7f6432df9640 1 --2- 172.21.15.194:0/320230436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f642c06d470 0x7f642c0b76c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f642000e9b0 tx=0x7f642000ee70 comp rx=0 tx=0).stop 2024-08-09T14:49:45.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.880+0000 7f6432df9640 1 -- 172.21.15.194:0/320230436 shutdown_connections 2024-08-09T14:49:45.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.880+0000 7f6432df9640 1 --2- 172.21.15.194:0/320230436 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f642c143270 0x7f642c1d8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.880+0000 7f6432df9640 1 --2- 172.21.15.194:0/320230436 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f642c0b4e50 0x7f642c1d7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.880+0000 7f6432df9640 1 --2- 172.21.15.194:0/320230436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f642c06d470 0x7f642c0b76c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.880+0000 7f6432df9640 1 -- 172.21.15.194:0/320230436 >> 172.21.15.194:0/320230436 conn(0x7f642c0b8810 msgr2=0x7f642c13ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:45.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.880+0000 7f6432df9640 1 -- 172.21.15.194:0/320230436 shutdown_connections 2024-08-09T14:49:45.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.880+0000 7f6432df9640 1 -- 172.21.15.194:0/320230436 wait complete. 2024-08-09T14:49:45.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.880+0000 7f6432df9640 1 librados: shutdown 2024-08-09T14:49:45.884 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:45.881+0000 7f33aaa46040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-08-09T14:49:45.892 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 0f901e55-2ba8-4144-9802-3a109884254c 5 2024-08-09T14:49:45.934 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:45.932+0000 7f33aaa46040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-08-09T14:49:45.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.990+0000 7f68b3405640 10 monclient: get_monmap_and_config 2024-08-09T14:49:45.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.990+0000 7f68b3405640 10 monclient: build_initial_monmap 2024-08-09T14:49:45.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.990+0000 7f68b3405640 10 monclient: monmap: 2024-08-09T14:49:45.992 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:45.992 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:45.992 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:49:45.990958+0000 2024-08-09T14:49:45.992 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:49:45.990958+0000 2024-08-09T14:49:45.992 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:45.992 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:45.992 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:49:45.992 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:49:45.992 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:49:45.992 INFO:teuthology.orchestra.run.smithi194.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:49:45.992 INFO:teuthology.orchestra.run.smithi194.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:49:45.992 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:45.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.990+0000 7f68b3405640 1 Processor -- start 2024-08-09T14:49:45.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.990+0000 7f68b3405640 1 -- start start 2024-08-09T14:49:45.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.990+0000 7f68b3405640 10 monclient: init 2024-08-09T14:49:45.995 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:45.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-08-09T14:49:45.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f68ac06d470 0x7f68ac146150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:45.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 10 monclient(hunting): picked mon.noname-a con 0x7f68ac06d470 addr v2:172.21.15.155:3300/0 2024-08-09T14:49:45.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f68ac0b44a0 0x7f68ac13ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:45.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 10 monclient(hunting): picked mon.noname-c con 0x7f68ac0b44a0 addr v2:172.21.15.194:3300/0 2024-08-09T14:49:45.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68ac0b4e50 0x7f68ac140450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:45.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 10 monclient(hunting): picked mon.noname-b con 0x7f68ac0b4e50 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:45.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:45.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f68ac06d850 con 0x7f68ac06d470 2024-08-09T14:49:45.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:45.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f68ac0b4930 con 0x7f68ac0b44a0 2024-08-09T14:49:45.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:45.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f68ac0b2e10 con 0x7f68ac0b4e50 2024-08-09T14:49:45.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:45.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.991+0000 7f68b3405640 10 monclient(hunting): authenticate will time out at 1211.025531s 2024-08-09T14:49:45.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b09a0640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f68ac0b44a0 0x7f68ac13ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:45.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b19a2640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68ac0b4e50 0x7f68ac140450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:45.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b11a1640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f68ac06d470 0x7f68ac146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:45.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b09a0640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f68ac0b44a0 0x7f68ac13ff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:46402/0 (socket says 172.21.15.194:46402) 2024-08-09T14:49:45.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b09a0640 1 -- 172.21.15.194:0/2924229371 learned_addr learned my addr 172.21.15.194:0/2924229371 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-08-09T14:49:45.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b09a0640 10 monclient(hunting): get_auth_request con 0x7f68ac0b44a0 auth_method 0 2024-08-09T14:49:45.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b09a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:45.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b09a0640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:45.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b09a0640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:45.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b19a2640 10 monclient(hunting): get_auth_request con 0x7f68ac0b4e50 auth_method 0 2024-08-09T14:49:45.997 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b19a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:45.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b19a2640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:45.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b19a2640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:45.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b11a1640 10 monclient(hunting): get_auth_request con 0x7f68ac06d470 auth_method 0 2024-08-09T14:49:45.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b11a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:45.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b11a1640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:45.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.992+0000 7f68b11a1640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:45.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.993+0000 7f68b19a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:45.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.993+0000 7f68b19a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:45.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.993+0000 7f68b19a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:45.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.993+0000 7f68b09a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:45.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.993+0000 7f68b09a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:45.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.993+0000 7f68b09a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:45.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.993+0000 7f68b11a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:45.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.993+0000 7f68b11a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:45.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.993+0000 7f68b11a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:45.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.994+0000 7f68b19a2640 10 monclient(hunting): handle_auth_done global_id 4130 payload 274 2024-08-09T14:49:45.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.994+0000 7f68b19a2640 1 -- 172.21.15.194:0/2924229371 >> v2:172.21.15.194:3300/0 conn(0x7f68ac0b44a0 msgr2=0x7f68ac13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:45.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.994+0000 7f68b19a2640 1 --2- 172.21.15.194:0/2924229371 >> v2:172.21.15.194:3300/0 conn(0x7f68ac0b44a0 0x7f68ac13ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.994+0000 7f68b19a2640 1 -- 172.21.15.194:0/2924229371 >> v2:172.21.15.155:3300/0 conn(0x7f68ac06d470 msgr2=0x7f68ac146150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:45.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.994+0000 7f68b19a2640 1 --2- 172.21.15.194:0/2924229371 >> v2:172.21.15.155:3300/0 conn(0x7f68ac06d470 0x7f68ac146150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:45.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.994+0000 7f68b19a2640 10 monclient: _finish_hunting 0 2024-08-09T14:49:45.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.994+0000 7f68b19a2640 1 monclient: found mon.noname-b 2024-08-09T14:49:45.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.994+0000 7f68b19a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:45.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.994+0000 7f68b19a2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:49:45.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.994+0000 7f68b19a2640 1 -- 172.21.15.194:0/2924229371 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68ac146690 con 0x7f68ac0b4e50 2024-08-09T14:49:46.000 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:45.996+0000 7f68b19a2640 1 --2- 172.21.15.194:0/2924229371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68ac0b4e50 0x7f68ac140450 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f68ac143d10 tx=0x7f68a402be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68bbc74add44b05b server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:46.001 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68a3fff640 1 -- 172.21.15.194:0/2924229371 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a4039070 con 0x7f68ac0b4e50 2024-08-09T14:49:46.002 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:46.002 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68a3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:49:46.002 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68a3fff640 10 monclient: dump: 2024-08-09T14:49:46.002 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:46.002 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:46.002 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.002 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.003 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:46.003 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.003 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:46.003 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:46.003 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:46.003 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68a3fff640 10 monclient: _finish_auth 0 2024-08-09T14:49:46.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68a3fff640 10 monclient: _check_auth_tickets 2024-08-09T14:49:46.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:46.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68a3fff640 1 -- 172.21.15.194:0/2924229371 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68a402cbd0 con 0x7f68ac0b4e50 2024-08-09T14:49:46.004 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68a3fff640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:46.004 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68a3fff640 1 -- 172.21.15.194:0/2924229371 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a4034440 con 0x7f68ac0b4e50 2024-08-09T14:49:46.004 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:46.004 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:49:46.004 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68a3fff640 10 monclient: dump: 2024-08-09T14:49:46.004 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:46.004 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:46.004 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.004 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.004 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:46.004 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.004 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:46.004 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:46.004 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:46.005 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.005 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68b3405640 5 monclient: authenticate success, global_id 4130 2024-08-09T14:49:46.005 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68b3405640 10 monclient: get_monmap_and_config success 2024-08-09T14:49:46.005 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.000+0000 7f68b3405640 4 set_mon_vals no callback set 2024-08-09T14:49:46.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.001+0000 7f68b3405640 10 monclient: shutdown 2024-08-09T14:49:46.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.001+0000 7f68b3405640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:46.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.001+0000 7f68b3405640 1 -- 172.21.15.194:0/2924229371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68ac0b4e50 msgr2=0x7f68ac140450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:46.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.001+0000 7f68b3405640 1 --2- 172.21.15.194:0/2924229371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68ac0b4e50 0x7f68ac140450 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f68ac143d10 tx=0x7f68a402be20 comp rx=0 tx=0).stop 2024-08-09T14:49:46.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.001+0000 7f68b3405640 1 -- 172.21.15.194:0/2924229371 shutdown_connections 2024-08-09T14:49:46.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.001+0000 7f68b3405640 1 --2- 172.21.15.194:0/2924229371 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68ac0b4e50 0x7f68ac140450 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.001+0000 7f68b3405640 1 --2- 172.21.15.194:0/2924229371 >> v2:172.21.15.194:3300/0 conn(0x7f68ac0b44a0 0x7f68ac13ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.001+0000 7f68b3405640 1 --2- 172.21.15.194:0/2924229371 >> v2:172.21.15.155:3300/0 conn(0x7f68ac06d470 0x7f68ac146150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.001+0000 7f68b3405640 1 -- 172.21.15.194:0/2924229371 >> 172.21.15.194:0/2924229371 conn(0x7f68ac0b8810 msgr2=0x7f68ac0b8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:46.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.001+0000 7f68b3405640 1 -- 172.21.15.194:0/2924229371 shutdown_connections 2024-08-09T14:49:46.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.001+0000 7f68b3405640 1 -- 172.21.15.194:0/2924229371 wait complete. 2024-08-09T14:49:46.009 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.002+0000 7f68b3405640 10 monclient: build_initial_monmap 2024-08-09T14:49:46.010 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.002+0000 7f68b3405640 10 monclient: monmap: 2024-08-09T14:49:46.010 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:46.010 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:49:46.010 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-08-09T14:49:46.010 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-08-09T14:49:46.010 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:46.010 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.010 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:49:46.010 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:49:46.010 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:49:46.010 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.010 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.002+0000 7f68b3405640 1 librados: starting msgr at 2024-08-09T14:49:46.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.002+0000 7f68b3405640 1 librados: starting objecter 2024-08-09T14:49:46.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.002+0000 7f68b3405640 1 Processor -- start 2024-08-09T14:49:46.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.002+0000 7f68b3405640 1 -- start start 2024-08-09T14:49:46.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.002+0000 7f68b3405640 1 librados: setting wanted keys 2024-08-09T14:49:46.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.002+0000 7f68b3405640 1 librados: calling monclient init 2024-08-09T14:49:46.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.002+0000 7f68b3405640 10 monclient: init 2024-08-09T14:49:46.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:46.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 10 monclient: _add_conns ranks=[0,1,2] 2024-08-09T14:49:46.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68ac06d470 0x7f68ac0b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:46.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 10 monclient(hunting): picked mon.noname-a con 0x7f68ac06d470 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:46.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68ac0b4e50 0x7f68ac13c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:46.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 10 monclient(hunting): picked mon.noname-b con 0x7f68ac0b4e50 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:49:46.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68ac13ceb0 0x7f68ac0ad4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:46.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 10 monclient(hunting): picked mon.noname-c con 0x7f68ac13ceb0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:46.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:46.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac0aad80 con 0x7f68ac06d470 2024-08-09T14:49:46.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:46.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f68ac0aac00 con 0x7f68ac0b4e50 2024-08-09T14:49:46.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:46.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f68ac0b3210 con 0x7f68ac13ceb0 2024-08-09T14:49:46.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:46.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b3405640 10 monclient(hunting): authenticate will time out at 1211.037326s 2024-08-09T14:49:46.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b09a0640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68ac0b4e50 0x7f68ac13c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:46.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b19a2640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68ac13ceb0 0x7f68ac0ad4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:46.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.003+0000 7f68b11a1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68ac06d470 0x7f68ac0b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:46.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b19a2640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68ac13ceb0 0x7f68ac0ad4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.194:35826/0 (socket says 172.21.15.194:35826) 2024-08-09T14:49:46.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b19a2640 1 -- 172.21.15.194:0/1626497897 learned_addr learned my addr 172.21.15.194:0/1626497897 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-08-09T14:49:46.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b19a2640 10 monclient(hunting): get_auth_request con 0x7f68ac13ceb0 auth_method 0 2024-08-09T14:49:46.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b19a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:46.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b19a2640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:46.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b19a2640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:46.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b09a0640 10 monclient(hunting): get_auth_request con 0x7f68ac0b4e50 auth_method 0 2024-08-09T14:49:46.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b09a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:46.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b09a0640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:46.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b09a0640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:46.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b11a1640 10 monclient(hunting): get_auth_request con 0x7f68ac06d470 auth_method 0 2024-08-09T14:49:46.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b11a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:46.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b11a1640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:46.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b11a1640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:46.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b19a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:46.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b19a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:46.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.004+0000 7f68b19a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:46.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.005+0000 7f68b09a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:46.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.005+0000 7f68b09a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:46.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.005+0000 7f68b09a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:46.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.005+0000 7f68b11a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:46.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.005+0000 7f68b11a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:46.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.005+0000 7f68b11a1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:46.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.007+0000 7f68b19a2640 10 monclient(hunting): handle_auth_done global_id 4133 payload 824 2024-08-09T14:49:46.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.007+0000 7f68b19a2640 1 -- 172.21.15.194:0/1626497897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68ac0b4e50 msgr2=0x7f68ac13c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:46.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.007+0000 7f68b19a2640 1 --2- 172.21.15.194:0/1626497897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68ac0b4e50 0x7f68ac13c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.007+0000 7f68b19a2640 1 -- 172.21.15.194:0/1626497897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68ac06d470 msgr2=0x7f68ac0b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:46.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.007+0000 7f68b19a2640 1 --2- 172.21.15.194:0/1626497897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68ac06d470 0x7f68ac0b0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.007+0000 7f68b19a2640 10 monclient: _finish_hunting 0 2024-08-09T14:49:46.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.007+0000 7f68b19a2640 1 monclient: found mon.noname-c 2024-08-09T14:49:46.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.007+0000 7f68b19a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:46.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.007+0000 7f68b19a2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:49:46.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.007+0000 7f68b19a2640 1 -- 172.21.15.194:0/1626497897 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68ac0b1b80 con 0x7f68ac13ceb0 2024-08-09T14:49:46.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.010+0000 7f68b19a2640 1 --2- 172.21.15.194:0/1626497897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68ac13ceb0 0x7f68ac0ad4b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f68a402c430 tx=0x7f68a4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:46.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.010+0000 7f68a1ffb640 10 client.?.objecter ms_handle_connect 0x7f68ac13ceb0 2024-08-09T14:49:46.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.010+0000 7f68a1ffb640 10 client.?.objecter resend_mon_ops 2024-08-09T14:49:46.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68a1ffb640 1 -- 172.21.15.194:0/1626497897 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a4044020 con 0x7f68ac13ceb0 2024-08-09T14:49:46.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:46.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68a1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:49:46.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68a1ffb640 10 monclient: dump: 2024-08-09T14:49:46.017 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:46.017 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:46.017 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.017 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.017 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:46.017 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.017 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:46.017 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:46.017 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:46.017 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68a1ffb640 10 monclient: _finish_auth 0 2024-08-09T14:49:46.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68a1ffb640 10 monclient: _check_auth_tickets 2024-08-09T14:49:46.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:46.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68a1ffb640 1 -- 172.21.15.194:0/1626497897 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68a4035e50 con 0x7f68ac13ceb0 2024-08-09T14:49:46.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68a1ffb640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:46.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68a1ffb640 1 -- 172.21.15.194:0/1626497897 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68a4039040 con 0x7f68ac13ceb0 2024-08-09T14:49:46.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68b3405640 5 monclient: authenticate success, global_id 4133 2024-08-09T14:49:46.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68b3405640 10 monclient: _renew_subs 2024-08-09T14:49:46.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68b3405640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:46.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68b3405640 1 -- 172.21.15.194:0/1626497897 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68ac0b1e60 con 0x7f68ac13ceb0 2024-08-09T14:49:46.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68a37fe640 4 set_mon_vals no callback set 2024-08-09T14:49:46.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:46.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68a1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:49:46.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.013+0000 7f68a1ffb640 10 monclient: dump: 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.014+0000 7f68b3405640 10 client.4133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.014+0000 7f68b3405640 10 monclient: _renew_subs 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.014+0000 7f68b3405640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.014+0000 7f68b3405640 1 -- 172.21.15.194:0/1626497897 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68ac13d340 con 0x7f68ac13ceb0 2024-08-09T14:49:46.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.014+0000 7f68b3405640 1 librados: init done 2024-08-09T14:49:46.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.014+0000 7f68b3405640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:46.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.014+0000 7f68b3405640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:46.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.014+0000 7f68b3405640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:46.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.014+0000 7f68b3405640 1 -- 172.21.15.194:0/1626497897 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68ac146150 con 0x7f68ac13ceb0 2024-08-09T14:49:46.025 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:46.024+0000 7f2c8b84e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-09T14:49:46.100 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:46.098+0000 7f33aaa46040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-08-09T14:49:46.111 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.110+0000 7f68a1ffb640 1 -- 172.21.15.194:0/1626497897 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f68a4047020 con 0x7f68ac13ceb0 2024-08-09T14:49:46.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.114+0000 7f68a1ffb640 1 -- 172.21.15.194:0/1626497897 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f68a4040420 con 0x7f68ac13ceb0 2024-08-09T14:49:46.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.114+0000 7f68a1ffb640 10 client.4133.objecter ms_dispatch 0x7f68ac000b90 osd_map(7..7 src has 1..7) 2024-08-09T14:49:46.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.114+0000 7f68a1ffb640 3 client.4133.objecter handle_osd_map got epochs [7,7] > 0 2024-08-09T14:49:46.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.114+0000 7f68a1ffb640 3 client.4133.objecter handle_osd_map decoding full epoch 7 2024-08-09T14:49:46.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.114+0000 7f68a1ffb640 20 client.4133.objecter dump_active .. 0 homeless 2024-08-09T14:49:46.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.114+0000 7f68a1ffb640 1 -- 172.21.15.194:0/1626497897 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7f68a404e030 con 0x7f68ac13ceb0 2024-08-09T14:49:46.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.114+0000 7f68a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:46.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.114+0000 7f68a1ffb640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:49:46.164 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:46.163+0000 7f2c8b84e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-09T14:49:46.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.189+0000 7f68b3405640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0f901e55-2ba8-4144-9802-3a109884254c", "id": 5}] 2024-08-09T14:49:46.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.190+0000 7f68b3405640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0f901e55-2ba8-4144-9802-3a109884254c", "id": 5}] 2024-08-09T14:49:46.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.190+0000 7f68b3405640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:46.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.190+0000 7f68b3405640 1 -- 172.21.15.194:0/1626497897 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0f901e55-2ba8-4144-9802-3a109884254c", "id": 5} v 0) -- 0x7f68ac13ee70 con 0x7f68ac13ceb0 2024-08-09T14:49:46.215 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:46.214+0000 7f2c8b84e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-09T14:49:46.239 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:46.238+0000 7f33aaa46040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-08-09T14:49:46.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.272+0000 7f68a1ffb640 1 -- 172.21.15.194:0/1626497897 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0f901e55-2ba8-4144-9802-3a109884254c", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f68a403fc20 con 0x7f68ac13ceb0 2024-08-09T14:49:46.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.272+0000 7f68a1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0f901e55-2ba8-4144-9802-3a109884254c", "id": 5}] 2024-08-09T14:49:46.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.272+0000 7f68a1ffb640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:49:46.273 INFO:teuthology.orchestra.run.smithi194.stdout:5 2024-08-09T14:49:46.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.272+0000 7f68b3405640 10 librados: watch_flush enter 2024-08-09T14:49:46.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 10 librados: watch_flush exit 2024-08-09T14:49:46.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 20 client.4133.objecter shutdown clearing up homeless session... 2024-08-09T14:49:46.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 10 client.4133.objecter successfully canceled tick 2024-08-09T14:49:46.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 10 monclient: shutdown 2024-08-09T14:49:46.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:46.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 1 -- 172.21.15.194:0/1626497897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68ac13ceb0 msgr2=0x7f68ac0ad4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:46.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 1 --2- 172.21.15.194:0/1626497897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68ac13ceb0 0x7f68ac0ad4b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f68a402c430 tx=0x7f68a4004a40 comp rx=0 tx=0).stop 2024-08-09T14:49:46.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 1 -- 172.21.15.194:0/1626497897 shutdown_connections 2024-08-09T14:49:46.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 1 --2- 172.21.15.194:0/1626497897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f68ac13ceb0 0x7f68ac0ad4b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 1 --2- 172.21.15.194:0/1626497897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68ac0b4e50 0x7f68ac13c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 1 --2- 172.21.15.194:0/1626497897 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f68ac06d470 0x7f68ac0b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 1 -- 172.21.15.194:0/1626497897 >> 172.21.15.194:0/1626497897 conn(0x7f68ac0b8810 msgr2=0x7f68ac0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:46.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 1 -- 172.21.15.194:0/1626497897 shutdown_connections 2024-08-09T14:49:46.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 1 -- 172.21.15.194:0/1626497897 wait complete. 2024-08-09T14:49:46.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.273+0000 7f68b3405640 1 librados: shutdown 2024-08-09T14:49:46.285 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new fd4b8215-814e-4b5a-9877-0ab63818b0ba 6 2024-08-09T14:49:46.292 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:46.291+0000 7f33aaa46040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-08-09T14:49:46.296 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:46.295+0000 7f2c8b84e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-09T14:49:46.345 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:46.344+0000 7f2c8b84e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-09T14:49:46.368 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:46.367+0000 7f33aaa46040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-08-09T14:49:46.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.391+0000 7fe08bce9640 10 monclient: get_monmap_and_config 2024-08-09T14:49:46.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.391+0000 7fe08bce9640 10 monclient: build_initial_monmap 2024-08-09T14:49:46.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.391+0000 7fe08bce9640 10 monclient: monmap: 2024-08-09T14:49:46.392 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:46.392 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:46.392 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:49:46.392108+0000 2024-08-09T14:49:46.392 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:49:46.392108+0000 2024-08-09T14:49:46.393 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:46.393 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.393 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:49:46.393 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:49:46.393 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:49:46.393 INFO:teuthology.orchestra.run.smithi194.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:49:46.393 INFO:teuthology.orchestra.run.smithi194.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:49:46.393 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.395 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.391+0000 7fe08bce9640 1 Processor -- start 2024-08-09T14:49:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 1 -- start start 2024-08-09T14:49:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 10 monclient: init 2024-08-09T14:49:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-08-09T14:49:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe08406d470 0x7fe084146150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 10 monclient(hunting): picked mon.noname-a con 0x7fe08406d470 addr v2:172.21.15.155:3300/0 2024-08-09T14:49:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fe0840b44a0 0x7fe08413ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 10 monclient(hunting): picked mon.noname-c con 0x7fe0840b44a0 addr v2:172.21.15.194:3300/0 2024-08-09T14:49:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe0840b4e50 addr v1:172.21.15.155:6789/0 2024-08-09T14:49:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0840b4930 con 0x7fe0840b4e50 2024-08-09T14:49:46.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7fe08406d850 con 0x7fe08406d470 2024-08-09T14:49:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7fe0840b2e10 con 0x7fe0840b44a0 2024-08-09T14:49:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.392+0000 7fe08bce9640 10 monclient(hunting): authenticate will time out at 1211.426670s 2024-08-09T14:49:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.393+0000 7fe089284640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fe0840b44a0 0x7fe08413ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.393+0000 7fe089a85640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fe08406d470 0x7fe084146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.393+0000 7fe08a286640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fe0840b4e50 0x7fe0840b3210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.194:45874/0 (socket says 172.21.15.194:45874) 2024-08-09T14:49:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.393+0000 7fe08a286640 1 -- 172.21.15.194:0/427910583 learned_addr learned my addr 172.21.15.194:0/427910583 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-08-09T14:49:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.393+0000 7fe089284640 10 monclient(hunting): get_auth_request con 0x7fe0840b44a0 auth_method 0 2024-08-09T14:49:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.393+0000 7fe089284640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:46.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.393+0000 7fe089284640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:46.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.393+0000 7fe089284640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:46.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.394+0000 7fe089a85640 10 monclient(hunting): get_auth_request con 0x7fe08406d470 auth_method 0 2024-08-09T14:49:46.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.394+0000 7fe089a85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:46.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.394+0000 7fe089a85640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:46.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.394+0000 7fe089a85640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:46.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.394+0000 7fe089284640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:46.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.394+0000 7fe089284640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:46.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.394+0000 7fe089284640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:46.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.395+0000 7fe089a85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:46.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.395+0000 7fe089a85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:46.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.395+0000 7fe089a85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:46.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.396+0000 7fe089284640 10 monclient(hunting): handle_auth_done global_id 4138 payload 274 2024-08-09T14:49:46.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.397+0000 7fe089284640 1 -- 172.21.15.194:0/427910583 >> v2:172.21.15.155:3300/0 conn(0x7fe08406d470 msgr2=0x7fe084146150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:46.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.397+0000 7fe089284640 1 --2- 172.21.15.194:0/427910583 >> v2:172.21.15.155:3300/0 conn(0x7fe08406d470 0x7fe084146150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.397+0000 7fe089284640 1 -- 172.21.15.194:0/427910583 >> v1:172.21.15.155:6789/0 conn(0x7fe0840b4e50 legacy=0x7fe0840b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:46.400 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.397+0000 7fe089284640 10 monclient: _finish_hunting 0 2024-08-09T14:49:46.400 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.397+0000 7fe089284640 1 monclient: found mon.noname-c 2024-08-09T14:49:46.400 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.397+0000 7fe089284640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:46.400 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.397+0000 7fe089284640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:49:46.400 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.397+0000 7fe089284640 1 -- 172.21.15.194:0/427910583 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe084146690 con 0x7fe0840b44a0 2024-08-09T14:49:46.400 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.399+0000 7fe089284640 1 --2- 172.21.15.194:0/427910583 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0840b44a0 0x7fe08413ff10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe084143d10 tx=0x7fe07402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca020c4170e06f22 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:46.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe088a83640 1 -- 172.21.15.194:0/427910583 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe074035020 con 0x7fe0840b44a0 2024-08-09T14:49:46.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe088a83640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:46.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe088a83640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:49:46.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe088a83640 10 monclient: dump: 2024-08-09T14:49:46.406 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:46.406 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:46.406 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.406 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.406 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:46.406 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.406 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:46.406 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:46.406 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:46.407 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe088a83640 10 monclient: _finish_auth 0 2024-08-09T14:49:46.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe088a83640 10 monclient: _check_auth_tickets 2024-08-09T14:49:46.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe088a83640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:46.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe088a83640 1 -- 172.21.15.194:0/427910583 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe074031830 con 0x7fe0840b44a0 2024-08-09T14:49:46.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe08bce9640 5 monclient: authenticate success, global_id 4138 2024-08-09T14:49:46.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe08bce9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:49:46.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe088a83640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:46.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe08bce9640 10 monclient: get_monmap_and_config success 2024-08-09T14:49:46.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe088a83640 1 -- 172.21.15.194:0/427910583 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe074031c00 con 0x7fe0840b44a0 2024-08-09T14:49:46.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe088a83640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:46.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe08bce9640 4 set_mon_vals no callback set 2024-08-09T14:49:46.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe088a83640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:49:46.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.403+0000 7fe088a83640 10 monclient: dump: 2024-08-09T14:49:46.408 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:46.408 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:46.408 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.408 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.408 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:46.408 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.408 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:46.408 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:46.408 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:46.408 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.404+0000 7fe08bce9640 10 monclient: shutdown 2024-08-09T14:49:46.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.404+0000 7fe08bce9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:46.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.404+0000 7fe08bce9640 1 -- 172.21.15.194:0/427910583 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0840b44a0 msgr2=0x7fe08413ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:46.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.404+0000 7fe08bce9640 1 --2- 172.21.15.194:0/427910583 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0840b44a0 0x7fe08413ff10 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe084143d10 tx=0x7fe07402b8d0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.404+0000 7fe08bce9640 1 -- 172.21.15.194:0/427910583 shutdown_connections 2024-08-09T14:49:46.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.404+0000 7fe08bce9640 1 --2- 172.21.15.194:0/427910583 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0840b44a0 0x7fe08413ff10 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.404+0000 7fe08bce9640 1 --2- 172.21.15.194:0/427910583 >> v2:172.21.15.155:3300/0 conn(0x7fe08406d470 0x7fe084146150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.404+0000 7fe08bce9640 1 -- 172.21.15.194:0/427910583 >> 172.21.15.194:0/427910583 conn(0x7fe0840b8810 msgr2=0x7fe0840b8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:46.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.404+0000 7fe08bce9640 1 -- 172.21.15.194:0/427910583 shutdown_connections 2024-08-09T14:49:46.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.404+0000 7fe08bce9640 1 -- 172.21.15.194:0/427910583 wait complete. 2024-08-09T14:49:46.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.405+0000 7fe08bce9640 10 monclient: build_initial_monmap 2024-08-09T14:49:46.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.405+0000 7fe08bce9640 10 monclient: monmap: 2024-08-09T14:49:46.412 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:46.412 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:49:46.413 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-08-09T14:49:46.413 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-08-09T14:49:46.413 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:46.413 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.413 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:49:46.413 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:49:46.413 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:49:46.413 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.405+0000 7fe08bce9640 1 librados: starting msgr at 2024-08-09T14:49:46.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.405+0000 7fe08bce9640 1 librados: starting objecter 2024-08-09T14:49:46.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.405+0000 7fe08bce9640 1 Processor -- start 2024-08-09T14:49:46.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.405+0000 7fe08bce9640 1 -- start start 2024-08-09T14:49:46.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.405+0000 7fe08bce9640 1 librados: setting wanted keys 2024-08-09T14:49:46.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.405+0000 7fe08bce9640 1 librados: calling monclient init 2024-08-09T14:49:46.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.405+0000 7fe08bce9640 10 monclient: init 2024-08-09T14:49:46.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:46.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 10 monclient: _add_conns ranks=[0,1,2] 2024-08-09T14:49:46.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe08406d470 0x7fe0841cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:46.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 10 monclient(hunting): picked mon.noname-a con 0x7fe08406d470 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:46.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0840b4e50 0x7fe084136aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:46.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 10 monclient(hunting): picked mon.noname-b con 0x7fe0840b4e50 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:49:46.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe084136fe0 0x7fe0841447e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:46.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 10 monclient(hunting): picked mon.noname-c con 0x7fe084136fe0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:46.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:46.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fe084140450 con 0x7fe08406d470 2024-08-09T14:49:46.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:46.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe0840aac40 con 0x7fe0840b4e50 2024-08-09T14:49:46.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:46.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fe0841405d0 con 0x7fe084136fe0 2024-08-09T14:49:46.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:46.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.406+0000 7fe08bce9640 10 monclient(hunting): authenticate will time out at 1211.440666s 2024-08-09T14:49:46.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089284640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0840b4e50 0x7fe084136aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:46.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089a85640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe08406d470 0x7fe0841cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:46.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe08a286640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe084136fe0 0x7fe0841447e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:46.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089284640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0840b4e50 0x7fe084136aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:46422/0 (socket says 172.21.15.194:46422) 2024-08-09T14:49:46.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089284640 1 -- 172.21.15.194:0/212184846 learned_addr learned my addr 172.21.15.194:0/212184846 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-08-09T14:49:46.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089284640 10 monclient(hunting): get_auth_request con 0x7fe0840b4e50 auth_method 0 2024-08-09T14:49:46.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089284640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:46.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089284640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:46.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089284640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:46.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089a85640 10 monclient(hunting): get_auth_request con 0x7fe08406d470 auth_method 0 2024-08-09T14:49:46.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089a85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:46.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089a85640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:46.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089a85640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:46.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe08a286640 10 monclient(hunting): get_auth_request con 0x7fe084136fe0 auth_method 0 2024-08-09T14:49:46.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe08a286640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:46.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe08a286640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:46.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe08a286640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:46.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089284640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:46.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089284640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:46.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.407+0000 7fe089284640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:46.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.408+0000 7fe08a286640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:46.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.408+0000 7fe08a286640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:46.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.408+0000 7fe08a286640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:46.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.408+0000 7fe089a85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:46.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.408+0000 7fe089a85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:46.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.408+0000 7fe089a85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:46.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.410+0000 7fe089284640 10 monclient(hunting): handle_auth_done global_id 4141 payload 824 2024-08-09T14:49:46.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.410+0000 7fe089284640 1 -- 172.21.15.194:0/212184846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe084136fe0 msgr2=0x7fe0841447e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:46.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.410+0000 7fe089284640 1 --2- 172.21.15.194:0/212184846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe084136fe0 0x7fe0841447e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.410+0000 7fe089284640 1 -- 172.21.15.194:0/212184846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe08406d470 msgr2=0x7fe0841cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:46.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.410+0000 7fe089284640 1 --2- 172.21.15.194:0/212184846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe08406d470 0x7fe0841cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.410+0000 7fe089284640 10 monclient: _finish_hunting 0 2024-08-09T14:49:46.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.410+0000 7fe089284640 1 monclient: found mon.noname-b 2024-08-09T14:49:46.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.410+0000 7fe089284640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:46.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.410+0000 7fe089284640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:49:46.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.410+0000 7fe089284640 1 -- 172.21.15.194:0/212184846 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0841cee20 con 0x7fe0840b4e50 2024-08-09T14:49:46.419 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.412+0000 7fe089284640 1 --2- 172.21.15.194:0/212184846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0840b4e50 0x7fe084136aa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe07402bee0 tx=0x7fe0740046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:46.419 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.413+0000 7fe07a7fc640 10 client.?.objecter ms_handle_connect 0x7fe0840b4e50 2024-08-09T14:49:46.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.413+0000 7fe07a7fc640 10 client.?.objecter resend_mon_ops 2024-08-09T14:49:46.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe07a7fc640 1 -- 172.21.15.194:0/212184846 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe074035040 con 0x7fe0840b4e50 2024-08-09T14:49:46.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe07a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:46.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe07a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:49:46.420 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe07a7fc640 10 monclient: dump: 2024-08-09T14:49:46.420 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:46.420 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:46.420 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.420 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.420 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:46.420 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.420 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:46.420 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:46.420 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:46.421 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe07a7fc640 10 monclient: _finish_auth 0 2024-08-09T14:49:46.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe07a7fc640 10 monclient: _check_auth_tickets 2024-08-09T14:49:46.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe07a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:46.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe08bce9640 5 monclient: authenticate success, global_id 4141 2024-08-09T14:49:46.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe07a7fc640 1 -- 172.21.15.194:0/212184846 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe07403f070 con 0x7fe0840b4e50 2024-08-09T14:49:46.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe07a7fc640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:46.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe07a7fc640 1 -- 172.21.15.194:0/212184846 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07403bb40 con 0x7fe0840b4e50 2024-08-09T14:49:46.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe07a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:46.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe07a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:49:46.421 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe07a7fc640 10 monclient: dump: 2024-08-09T14:49:46.421 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:46.421 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:46.422 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.422 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.422 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:46.422 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.422 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:46.422 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:46.422 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:46.422 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.422 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe07bfff640 4 set_mon_vals no callback set 2024-08-09T14:49:46.422 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe08bce9640 10 monclient: _renew_subs 2024-08-09T14:49:46.422 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe08bce9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:46.422 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.416+0000 7fe08bce9640 1 -- 172.21.15.194:0/212184846 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe084140a50 con 0x7fe0840b4e50 2024-08-09T14:49:46.422 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.417+0000 7fe08bce9640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:49:46.423 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.417+0000 7fe08bce9640 10 monclient: _renew_subs 2024-08-09T14:49:46.423 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.417+0000 7fe08bce9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:46.423 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.417+0000 7fe08bce9640 1 -- 172.21.15.194:0/212184846 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe084137460 con 0x7fe0840b4e50 2024-08-09T14:49:46.423 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.417+0000 7fe08bce9640 1 librados: init done 2024-08-09T14:49:46.423 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.417+0000 7fe08bce9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:46.423 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.417+0000 7fe08bce9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:46.423 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.417+0000 7fe08bce9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:46.423 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.417+0000 7fe08bce9640 1 -- 172.21.15.194:0/212184846 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe084146150 con 0x7fe0840b4e50 2024-08-09T14:49:46.423 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:46.418+0000 7f33aaa46040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-08-09T14:49:46.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.420+0000 7fe07a7fc640 1 -- 172.21.15.194:0/212184846 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe074038020 con 0x7fe0840b4e50 2024-08-09T14:49:46.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.420+0000 7fe07a7fc640 1 -- 172.21.15.194:0/212184846 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fe07403aa20 con 0x7fe0840b4e50 2024-08-09T14:49:46.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.420+0000 7fe07a7fc640 10 client.4141.objecter ms_dispatch 0x7fe084000b90 osd_map(8..8 src has 1..8) 2024-08-09T14:49:46.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.420+0000 7fe07a7fc640 3 client.4141.objecter handle_osd_map got epochs [8,8] > 0 2024-08-09T14:49:46.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.420+0000 7fe07a7fc640 3 client.4141.objecter handle_osd_map decoding full epoch 8 2024-08-09T14:49:46.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.420+0000 7fe07a7fc640 20 client.4141.objecter dump_active .. 0 homeless 2024-08-09T14:49:46.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.517+0000 7fe07a7fc640 1 -- 172.21.15.194:0/212184846 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7fe07403be20 con 0x7fe0840b4e50 2024-08-09T14:49:46.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.517+0000 7fe07a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:46.519 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.517+0000 7fe07a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:49:46.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.590+0000 7fe08bce9640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fd4b8215-814e-4b5a-9877-0ab63818b0ba", "id": 6}] 2024-08-09T14:49:46.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.590+0000 7fe08bce9640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fd4b8215-814e-4b5a-9877-0ab63818b0ba", "id": 6}] 2024-08-09T14:49:46.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.590+0000 7fe08bce9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:46.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.590+0000 7fe08bce9640 1 -- 172.21.15.194:0/212184846 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fd4b8215-814e-4b5a-9877-0ab63818b0ba", "id": 6} v 0) -- 0x7fe0840b74c0 con 0x7fe0840b4e50 2024-08-09T14:49:46.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.652+0000 7fe07a7fc640 1 -- 172.21.15.194:0/212184846 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fd4b8215-814e-4b5a-9877-0ab63818b0ba", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fe07403ac20 con 0x7fe0840b4e50 2024-08-09T14:49:46.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.652+0000 7fe07a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fd4b8215-814e-4b5a-9877-0ab63818b0ba", "id": 6}] 2024-08-09T14:49:46.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.652+0000 7fe07a7fc640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:49:46.653 INFO:teuthology.orchestra.run.smithi194.stdout:6 2024-08-09T14:49:46.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 10 librados: watch_flush enter 2024-08-09T14:49:46.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 10 librados: watch_flush exit 2024-08-09T14:49:46.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 20 client.4141.objecter shutdown clearing up homeless session... 2024-08-09T14:49:46.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 10 client.4141.objecter successfully canceled tick 2024-08-09T14:49:46.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 10 monclient: shutdown 2024-08-09T14:49:46.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:46.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 1 -- 172.21.15.194:0/212184846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0840b4e50 msgr2=0x7fe084136aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:46.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 1 --2- 172.21.15.194:0/212184846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0840b4e50 0x7fe084136aa0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe07402bee0 tx=0x7fe0740046c0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 1 -- 172.21.15.194:0/212184846 shutdown_connections 2024-08-09T14:49:46.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 1 --2- 172.21.15.194:0/212184846 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fe084136fe0 0x7fe0841447e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 1 --2- 172.21.15.194:0/212184846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0840b4e50 0x7fe084136aa0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 1 --2- 172.21.15.194:0/212184846 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fe08406d470 0x7fe0841cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 1 -- 172.21.15.194:0/212184846 >> 172.21.15.194:0/212184846 conn(0x7fe0840b8810 msgr2=0x7fe0840b6400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:46.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 1 -- 172.21.15.194:0/212184846 shutdown_connections 2024-08-09T14:49:46.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 1 -- 172.21.15.194:0/212184846 wait complete. 2024-08-09T14:49:46.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.653+0000 7fe08bce9640 1 librados: shutdown 2024-08-09T14:49:46.665 DEBUG:teuthology.orchestra.run.smithi194:> sudo ceph --cluster ceph osd new 26ed84b6-639d-4ccd-bb1d-c64b593ada7d 7 2024-08-09T14:49:46.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.763+0000 7f0bd784d640 10 monclient: get_monmap_and_config 2024-08-09T14:49:46.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.763+0000 7f0bd784d640 10 monclient: build_initial_monmap 2024-08-09T14:49:46.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.763+0000 7f0bd784d640 10 monclient: monmap: 2024-08-09T14:49:46.764 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:46.765 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:46.765 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:49:46.763943+0000 2024-08-09T14:49:46.765 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:49:46.763943+0000 2024-08-09T14:49:46.765 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:46.765 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.765 INFO:teuthology.orchestra.run.smithi194.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:49:46.765 INFO:teuthology.orchestra.run.smithi194.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:49:46.765 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:49:46.765 INFO:teuthology.orchestra.run.smithi194.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:49:46.765 INFO:teuthology.orchestra.run.smithi194.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:49:46.765 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.763+0000 7f0bd784d640 1 Processor -- start 2024-08-09T14:49:46.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.763+0000 7f0bd784d640 1 -- start start 2024-08-09T14:49:46.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.763+0000 7f0bd784d640 10 monclient: init 2024-08-09T14:49:46.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.764+0000 7f0bd784d640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:46.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.764+0000 7f0bd784d640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-08-09T14:49:46.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.764+0000 7f0bd784d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0bd006d470 addr v1:172.21.15.194:6789/0 2024-08-09T14:49:46.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.764+0000 7f0bd784d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0bd00b44a0 addr v1:172.21.15.155:6789/0 2024-08-09T14:49:46.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.764+0000 7f0bd784d640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0bd0140620 0x7f0bd014e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:46.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.764+0000 7f0bd784d640 10 monclient(hunting): picked mon.noname-a con 0x7f0bd0140620 addr v2:172.21.15.155:3300/0 2024-08-09T14:49:46.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.764+0000 7f0bd784d640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bd00b4930 con 0x7f0bd00b44a0 2024-08-09T14:49:46.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.764+0000 7f0bd784d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bd00b2e10 con 0x7f0bd006d470 2024-08-09T14:49:46.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.764+0000 7f0bd784d640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:46.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.764+0000 7f0bd784d640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0bd006d850 con 0x7f0bd0140620 2024-08-09T14:49:46.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.764+0000 7f0bd784d640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:46.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.764+0000 7f0bd784d640 10 monclient(hunting): authenticate will time out at 1211.798591s 2024-08-09T14:49:46.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.765+0000 7f0bd55e9640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f0bd006d470 0x7f0bd0146150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.194:49036/0 (socket says 172.21.15.194:49036) 2024-08-09T14:49:46.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.765+0000 7f0bd55e9640 1 -- 172.21.15.194:0/2139392353 learned_addr learned my addr 172.21.15.194:0/2139392353 (peer_addr_for_me v1:172.21.15.194:0/0) 2024-08-09T14:49:46.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.765+0000 7f0bd5dea640 1 --2- 172.21.15.194:0/2139392353 >> v2:172.21.15.155:3300/0 conn(0x7f0bd0140620 0x7f0bd014e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:46.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.765+0000 7f0bd5dea640 10 monclient(hunting): get_auth_request con 0x7f0bd0140620 auth_method 0 2024-08-09T14:49:46.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.765+0000 7f0bd5dea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:46.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.765+0000 7f0bd5dea640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:46.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.765+0000 7f0bd5dea640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:46.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.766+0000 7f0bd5dea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:46.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.766+0000 7f0bd5dea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:46.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.766+0000 7f0bd5dea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:46.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.767+0000 7f0bc7fff640 1 -- 172.21.15.194:0/2139392353 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2893205783 0 0) 0x7f0bd00b2e10 con 0x7f0bd006d470 2024-08-09T14:49:46.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.767+0000 7f0bc7fff640 10 monclient(hunting): _negotiate 2024-08-09T14:49:46.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.767+0000 7f0bc7fff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:46.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.767+0000 7f0bc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:46.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.767+0000 7f0bc7fff640 10 monclient(hunting): my global_id is 4144 2024-08-09T14:49:46.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.767+0000 7f0bc7fff640 1 -- 172.21.15.194:0/2139392353 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bbc003520 con 0x7f0bd006d470 2024-08-09T14:49:46.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.768+0000 7f0bd5dea640 10 monclient(hunting): handle_auth_done global_id 4146 payload 274 2024-08-09T14:49:46.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.768+0000 7f0bd5dea640 1 -- 172.21.15.194:0/2139392353 >> v1:172.21.15.194:6789/0 conn(0x7f0bd006d470 legacy=0x7f0bd0146150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:46.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.768+0000 7f0bd5dea640 1 -- 172.21.15.194:0/2139392353 >> v1:172.21.15.155:6789/0 conn(0x7f0bd00b44a0 legacy=0x7f0bd013ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:46.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.768+0000 7f0bd5dea640 10 monclient: _finish_hunting 0 2024-08-09T14:49:46.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.768+0000 7f0bd5dea640 1 monclient: found mon.noname-a 2024-08-09T14:49:46.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.768+0000 7f0bd5dea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:46.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.768+0000 7f0bd5dea640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:49:46.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.768+0000 7f0bd5dea640 1 -- 172.21.15.194:0/2139392353 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd0146860 con 0x7f0bd0140620 2024-08-09T14:49:46.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.771+0000 7f0bd5dea640 1 --2- 172.21.15.194:0/2139392353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd0140620 0x7f0bd014e490 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009d90 tx=0x7f0bc802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee16226726046d20 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:46.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bc7fff640 1 -- 172.21.15.194:0/2139392353 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8039070 con 0x7f0bd0140620 2024-08-09T14:49:46.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:46.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bc7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:49:46.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bc7fff640 10 monclient: dump: 2024-08-09T14:49:46.776 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:46.777 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:46.777 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.777 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.777 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:46.777 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.777 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:46.777 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:46.777 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:46.777 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bc7fff640 10 monclient: _finish_auth 0 2024-08-09T14:49:46.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bc7fff640 10 monclient: _check_auth_tickets 2024-08-09T14:49:46.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:46.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bc7fff640 1 -- 172.21.15.194:0/2139392353 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bc802cd00 con 0x7f0bd0140620 2024-08-09T14:49:46.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bc7fff640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:46.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bc7fff640 1 -- 172.21.15.194:0/2139392353 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bc8034660 con 0x7f0bd0140620 2024-08-09T14:49:46.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:46.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bc7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:49:46.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bc7fff640 10 monclient: dump: 2024-08-09T14:49:46.778 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:46.778 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:46.778 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.778 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.778 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:46.778 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.778 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:46.779 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:46.779 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:46.779 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bd784d640 5 monclient: authenticate success, global_id 4146 2024-08-09T14:49:46.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bd784d640 10 monclient: get_monmap_and_config success 2024-08-09T14:49:46.779 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.775+0000 7f0bd784d640 4 set_mon_vals no callback set 2024-08-09T14:49:46.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.776+0000 7f0bd784d640 10 monclient: shutdown 2024-08-09T14:49:46.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.776+0000 7f0bd784d640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:46.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.776+0000 7f0bd784d640 1 -- 172.21.15.194:0/2139392353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd0140620 msgr2=0x7f0bd014e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:46.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.776+0000 7f0bd784d640 1 --2- 172.21.15.194:0/2139392353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd0140620 0x7f0bd014e490 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0bc8009d90 tx=0x7f0bc802bf50 comp rx=0 tx=0).stop 2024-08-09T14:49:46.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.776+0000 7f0bd784d640 1 -- 172.21.15.194:0/2139392353 shutdown_connections 2024-08-09T14:49:46.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.776+0000 7f0bd784d640 1 --2- 172.21.15.194:0/2139392353 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd0140620 0x7f0bd014e490 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.776+0000 7f0bd784d640 1 -- 172.21.15.194:0/2139392353 >> 172.21.15.194:0/2139392353 conn(0x7f0bd00b8810 msgr2=0x7f0bd00b8c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:46.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.776+0000 7f0bd784d640 1 -- 172.21.15.194:0/2139392353 shutdown_connections 2024-08-09T14:49:46.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.776+0000 7f0bd784d640 1 -- 172.21.15.194:0/2139392353 wait complete. 2024-08-09T14:49:46.783 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.776+0000 7f0bd784d640 10 monclient: build_initial_monmap 2024-08-09T14:49:46.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.776+0000 7f0bd784d640 10 monclient: monmap: 2024-08-09T14:49:46.784 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 0 2024-08-09T14:49:46.784 INFO:teuthology.orchestra.run.smithi194.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:49:46.784 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 0.000000 2024-08-09T14:49:46.784 INFO:teuthology.orchestra.run.smithi194.stderr:created 0.000000 2024-08-09T14:49:46.784 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:46.784 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.784 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:49:46.784 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:49:46.784 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:49:46.784 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.777+0000 7f0bd784d640 1 librados: starting msgr at 2024-08-09T14:49:46.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.777+0000 7f0bd784d640 1 librados: starting objecter 2024-08-09T14:49:46.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.777+0000 7f0bd784d640 1 Processor -- start 2024-08-09T14:49:46.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.777+0000 7f0bd784d640 1 -- start start 2024-08-09T14:49:46.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.777+0000 7f0bd784d640 1 librados: setting wanted keys 2024-08-09T14:49:46.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.777+0000 7f0bd784d640 1 librados: calling monclient init 2024-08-09T14:49:46.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.777+0000 7f0bd784d640 10 monclient: init 2024-08-09T14:49:46.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.777+0000 7f0bd784d640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:46.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.777+0000 7f0bd784d640 10 monclient: _add_conns ranks=[2,1,0] 2024-08-09T14:49:46.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.777+0000 7f0bd784d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd006d470 0x7f0bd0145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:46.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.777+0000 7f0bd784d640 10 monclient(hunting): picked mon.noname-c con 0x7f0bd006d470 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:46.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.777+0000 7f0bd784d640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0bd0140620 0x7f0bd0140a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:46.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.777+0000 7f0bd784d640 10 monclient(hunting): picked mon.noname-b con 0x7f0bd0140620 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:49:46.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.777+0000 7f0bd784d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd0140f90 0x7f0bd01378b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:46.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd784d640 10 monclient(hunting): picked mon.noname-a con 0x7f0bd0140f90 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:46.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd784d640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:46.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd784d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd00aa9f0 con 0x7f0bd0140f90 2024-08-09T14:49:46.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd784d640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:46.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd784d640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd00b50a0 con 0x7f0bd0140620 2024-08-09T14:49:46.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd784d640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:46.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd784d640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0bd00aab70 con 0x7f0bd006d470 2024-08-09T14:49:46.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd784d640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:46.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd784d640 10 monclient(hunting): authenticate will time out at 1211.812031s 2024-08-09T14:49:46.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd4de8640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0bd0140620 0x7f0bd0140a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:46.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd55e9640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd006d470 0x7f0bd0145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:46.786 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd5dea640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd0140f90 0x7f0bd01378b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:46.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd4de8640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0bd0140620 0x7f0bd0140a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:46428/0 (socket says 172.21.15.194:46428) 2024-08-09T14:49:46.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd4de8640 1 -- 172.21.15.194:0/3605506122 learned_addr learned my addr 172.21.15.194:0/3605506122 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-08-09T14:49:46.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd4de8640 10 monclient(hunting): get_auth_request con 0x7f0bd0140620 auth_method 0 2024-08-09T14:49:46.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd4de8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:46.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd4de8640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:46.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.778+0000 7f0bd4de8640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:46.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd5dea640 10 monclient(hunting): get_auth_request con 0x7f0bd0140f90 auth_method 0 2024-08-09T14:49:46.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd5dea640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:46.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd5dea640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:46.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd5dea640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:46.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd55e9640 10 monclient(hunting): get_auth_request con 0x7f0bd006d470 auth_method 0 2024-08-09T14:49:46.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd55e9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:46.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd55e9640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:46.787 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd55e9640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:46.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd4de8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:46.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd4de8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:46.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd4de8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:46.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd5dea640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:46.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd5dea640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:46.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd5dea640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:46.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd55e9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:46.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd55e9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:46.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.779+0000 7f0bd55e9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:46.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.781+0000 7f0bd4de8640 10 monclient(hunting): handle_auth_done global_id 4147 payload 824 2024-08-09T14:49:46.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.782+0000 7f0bd4de8640 1 -- 172.21.15.194:0/3605506122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd006d470 msgr2=0x7f0bd0145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:46.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.782+0000 7f0bd4de8640 1 --2- 172.21.15.194:0/3605506122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd006d470 0x7f0bd0145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.782+0000 7f0bd4de8640 1 -- 172.21.15.194:0/3605506122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd0140f90 msgr2=0x7f0bd01378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:46.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.782+0000 7f0bd4de8640 1 --2- 172.21.15.194:0/3605506122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd0140f90 0x7f0bd01378b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:46.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.782+0000 7f0bd4de8640 10 monclient: _finish_hunting 0 2024-08-09T14:49:46.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.782+0000 7f0bd4de8640 1 monclient: found mon.noname-b 2024-08-09T14:49:46.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.782+0000 7f0bd4de8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:46.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.782+0000 7f0bd4de8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:49:46.789 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.782+0000 7f0bd4de8640 1 -- 172.21.15.194:0/3605506122 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bd01ccdf0 con 0x7f0bd0140620 2024-08-09T14:49:46.790 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.784+0000 7f0bd4de8640 1 --2- 172.21.15.194:0/3605506122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0bd0140620 0x7f0bd0140a50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc00c430 tx=0x7f0bcc00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:46.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.784+0000 7f0bc5ffb640 10 client.?.objecter ms_handle_connect 0x7f0bd0140620 2024-08-09T14:49:46.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.784+0000 7f0bc5ffb640 10 client.?.objecter resend_mon_ops 2024-08-09T14:49:46.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bc5ffb640 1 -- 172.21.15.194:0/3605506122 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc0052d0 con 0x7f0bd0140620 2024-08-09T14:49:46.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:46.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bc5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:49:46.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bc5ffb640 10 monclient: dump: 2024-08-09T14:49:46.791 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:46.791 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:46.791 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.791 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.791 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:46.791 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.791 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:46.792 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:46.792 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:46.792 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bc5ffb640 10 monclient: _finish_auth 0 2024-08-09T14:49:46.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bc5ffb640 10 monclient: _check_auth_tickets 2024-08-09T14:49:46.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:46.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bc5ffb640 1 -- 172.21.15.194:0/3605506122 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bcc005490 con 0x7f0bd0140620 2024-08-09T14:49:46.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bc5ffb640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:46.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bc5ffb640 1 -- 172.21.15.194:0/3605506122 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bcc007260 con 0x7f0bd0140620 2024-08-09T14:49:46.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bd784d640 5 monclient: authenticate success, global_id 4147 2024-08-09T14:49:46.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bd784d640 10 monclient: _renew_subs 2024-08-09T14:49:46.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bd784d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:46.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bd784d640 1 -- 172.21.15.194:0/3605506122 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bd0141420 con 0x7f0bd0140620 2024-08-09T14:49:46.793 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:46.793 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bc5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:49:46.793 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bc5ffb640 10 monclient: dump: 2024-08-09T14:49:46.793 INFO:teuthology.orchestra.run.smithi194.stderr:epoch 1 2024-08-09T14:49:46.793 INFO:teuthology.orchestra.run.smithi194.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:46.793 INFO:teuthology.orchestra.run.smithi194.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.793 INFO:teuthology.orchestra.run.smithi194.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:46.793 INFO:teuthology.orchestra.run.smithi194.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:46.793 INFO:teuthology.orchestra.run.smithi194.stderr:election_strategy: 1 2024-08-09T14:49:46.793 INFO:teuthology.orchestra.run.smithi194.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:46.793 INFO:teuthology.orchestra.run.smithi194.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:46.793 INFO:teuthology.orchestra.run.smithi194.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:46.794 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T14:49:46.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bc77fe640 4 set_mon_vals no callback set 2024-08-09T14:49:46.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bd784d640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:49:46.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bd784d640 10 monclient: _renew_subs 2024-08-09T14:49:46.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bd784d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:46.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bd784d640 1 -- 172.21.15.194:0/3605506122 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bd0143040 con 0x7f0bd0140620 2024-08-09T14:49:46.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.787+0000 7f0bd784d640 1 librados: init done 2024-08-09T14:49:46.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.788+0000 7f0bd784d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:46.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.788+0000 7f0bd784d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:46.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.788+0000 7f0bd784d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:46.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.788+0000 7f0bd784d640 1 -- 172.21.15.194:0/3605506122 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0bd0146e90 con 0x7f0bd0140620 2024-08-09T14:49:46.795 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.790+0000 7f0bc5ffb640 1 -- 172.21.15.194:0/3605506122 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0bcc007400 con 0x7f0bd0140620 2024-08-09T14:49:46.795 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.790+0000 7f0bc5ffb640 1 -- 172.21.15.194:0/3605506122 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f0bcc011bf0 con 0x7f0bd0140620 2024-08-09T14:49:46.795 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.790+0000 7f0bc5ffb640 10 client.4147.objecter ms_dispatch 0x7f0bd0000b90 osd_map(9..9 src has 1..9) 2024-08-09T14:49:46.795 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.790+0000 7f0bc5ffb640 3 client.4147.objecter handle_osd_map got epochs [9,9] > 0 2024-08-09T14:49:46.795 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.790+0000 7f0bc5ffb640 3 client.4147.objecter handle_osd_map decoding full epoch 9 2024-08-09T14:49:46.795 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.791+0000 7f0bc5ffb640 20 client.4147.objecter dump_active .. 0 homeless 2024-08-09T14:49:46.841 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:46.839+0000 7f2c8b84e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-09T14:49:46.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.888+0000 7f0bc5ffb640 1 -- 172.21.15.194:0/3605506122 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7f0bd0146e90 con 0x7f0bd0140620 2024-08-09T14:49:46.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.888+0000 7f0bc5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:46.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.888+0000 7f0bc5ffb640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:49:46.920 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:46.919+0000 7f33aaa46040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-08-09T14:49:46.957 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:46.955+0000 7f2c8b84e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-09T14:49:46.962 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.961+0000 7f0bd784d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "26ed84b6-639d-4ccd-bb1d-c64b593ada7d", "id": 7}] 2024-08-09T14:49:46.962 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.961+0000 7f0bd784d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "26ed84b6-639d-4ccd-bb1d-c64b593ada7d", "id": 7}] 2024-08-09T14:49:46.962 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.961+0000 7f0bd784d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:46.962 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:46.961+0000 7f0bd784d640 1 -- 172.21.15.194:0/3605506122 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd new", "uuid": "26ed84b6-639d-4ccd-bb1d-c64b593ada7d", "id": 7} v 0) -- 0x7f0bd0146150 con 0x7f0bd0140620 2024-08-09T14:49:47.038 INFO:tasks.ceph.mgr.x.smithi194.stderr:2024-08-09T14:49:47.037+0000 7f33aaa46040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-08-09T14:49:47.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.126+0000 7f0bc5ffb640 1 -- 172.21.15.194:0/3605506122 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "26ed84b6-639d-4ccd-bb1d-c64b593ada7d", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f0bcc020070 con 0x7f0bd0140620 2024-08-09T14:49:47.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.126+0000 7f0bc5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "26ed84b6-639d-4ccd-bb1d-c64b593ada7d", "id": 7}] 2024-08-09T14:49:47.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.126+0000 7f0bc5ffb640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:49:47.128 INFO:teuthology.orchestra.run.smithi194.stdout:7 2024-08-09T14:49:47.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.127+0000 7f0bd784d640 10 librados: watch_flush enter 2024-08-09T14:49:47.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.127+0000 7f0bd784d640 10 librados: watch_flush exit 2024-08-09T14:49:47.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.127+0000 7f0bd784d640 20 client.4147.objecter shutdown clearing up homeless session... 2024-08-09T14:49:47.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.127+0000 7f0bd784d640 10 client.4147.objecter successfully canceled tick 2024-08-09T14:49:47.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.127+0000 7f0bd784d640 10 monclient: shutdown 2024-08-09T14:49:47.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.127+0000 7f0bd784d640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:47.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.127+0000 7f0bd784d640 1 -- 172.21.15.194:0/3605506122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0bd0140620 msgr2=0x7f0bd0140a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:47.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.127+0000 7f0bd784d640 1 --2- 172.21.15.194:0/3605506122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0bd0140620 0x7f0bd0140a50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc00c430 tx=0x7f0bcc00c8f0 comp rx=0 tx=0).stop 2024-08-09T14:49:47.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.127+0000 7f0bd784d640 1 -- 172.21.15.194:0/3605506122 shutdown_connections 2024-08-09T14:49:47.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.127+0000 7f0bd784d640 1 --2- 172.21.15.194:0/3605506122 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bd0140f90 0x7f0bd01378b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:47.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.127+0000 7f0bd784d640 1 --2- 172.21.15.194:0/3605506122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0bd0140620 0x7f0bd0140a50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:47.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.127+0000 7f0bd784d640 1 --2- 172.21.15.194:0/3605506122 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bd006d470 0x7f0bd0145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:47.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.127+0000 7f0bd784d640 1 -- 172.21.15.194:0/3605506122 >> 172.21.15.194:0/3605506122 conn(0x7f0bd00b8810 msgr2=0x7f0bd00b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:47.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.128+0000 7f0bd784d640 1 -- 172.21.15.194:0/3605506122 shutdown_connections 2024-08-09T14:49:47.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.128+0000 7f0bd784d640 1 -- 172.21.15.194:0/3605506122 wait complete. 2024-08-09T14:49:47.131 INFO:teuthology.orchestra.run.smithi194.stderr:2024-08-09T14:49:47.128+0000 7f0bd784d640 1 librados: shutdown 2024-08-09T14:49:47.139 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-09T14:49:47.139 INFO:tasks.ceph.osd.0:Restarting daemon 2024-08-09T14:49:47.139 DEBUG:teuthology.orchestra.run.smithi155:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-08-09T14:49:47.142 INFO:tasks.ceph.osd.0:Started 2024-08-09T14:49:47.142 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-09T14:49:47.142 INFO:tasks.ceph.osd.1:Restarting daemon 2024-08-09T14:49:47.142 DEBUG:teuthology.orchestra.run.smithi155:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-08-09T14:49:47.144 INFO:tasks.ceph.osd.1:Started 2024-08-09T14:49:47.144 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-09T14:49:47.145 INFO:tasks.ceph.osd.2:Restarting daemon 2024-08-09T14:49:47.145 DEBUG:teuthology.orchestra.run.smithi155:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-08-09T14:49:47.147 INFO:tasks.ceph.osd.2:Started 2024-08-09T14:49:47.147 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-09T14:49:47.147 INFO:tasks.ceph.osd.3:Restarting daemon 2024-08-09T14:49:47.147 DEBUG:teuthology.orchestra.run.smithi155:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-08-09T14:49:47.150 INFO:tasks.ceph.osd.3:Started 2024-08-09T14:49:47.150 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-09T14:49:47.151 INFO:tasks.ceph.osd.4:Restarting daemon 2024-08-09T14:49:47.151 DEBUG:teuthology.orchestra.run.smithi194:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-08-09T14:49:47.153 INFO:tasks.ceph.osd.4:Started 2024-08-09T14:49:47.153 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-09T14:49:47.153 INFO:tasks.ceph.osd.5:Restarting daemon 2024-08-09T14:49:47.153 DEBUG:teuthology.orchestra.run.smithi194:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-08-09T14:49:47.155 INFO:tasks.ceph.osd.5:Started 2024-08-09T14:49:47.156 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-09T14:49:47.156 INFO:tasks.ceph.osd.6:Restarting daemon 2024-08-09T14:49:47.156 DEBUG:teuthology.orchestra.run.smithi194:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-08-09T14:49:47.158 INFO:tasks.ceph.osd.6:Started 2024-08-09T14:49:47.158 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-08-09T14:49:47.158 INFO:tasks.ceph.osd.7:Restarting daemon 2024-08-09T14:49:47.158 DEBUG:teuthology.orchestra.run.smithi194:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-08-09T14:49:47.160 INFO:tasks.ceph.osd.7:Started 2024-08-09T14:49:47.161 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-09T14:49:47.286 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.284+0000 7f0894b40640 10 monclient: get_monmap_and_config 2024-08-09T14:49:47.286 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.284+0000 7f0894b40640 10 monclient: build_initial_monmap 2024-08-09T14:49:47.286 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.285+0000 7f0894b40640 10 monclient: monmap: 2024-08-09T14:49:47.286 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:49:47.287 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:47.287 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:49:47.286009+0000 2024-08-09T14:49:47.287 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:49:47.286009+0000 2024-08-09T14:49:47.287 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:47.287 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:47.287 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:49:47.287 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:49:47.287 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:49:47.287 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:49:47.287 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:49:47.287 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:49:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.285+0000 7f0894b40640 1 Processor -- start 2024-08-09T14:49:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.285+0000 7f0894b40640 1 -- start start 2024-08-09T14:49:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.285+0000 7f0894b40640 10 monclient: init 2024-08-09T14:49:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.286+0000 7f0894b40640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.286+0000 7f0894b40640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-08-09T14:49:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.286+0000 7f0894b40640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08900b0160 0x7f08900b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.286+0000 7f0894b40640 10 monclient(hunting): picked mon.noname-b con 0x7f08900b0160 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.286+0000 7f0894b40640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f08900b22e0 0x7f08901469d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.286+0000 7f0894b40640 10 monclient(hunting): picked mon.noname-c con 0x7f08900b22e0 addr v2:172.21.15.194:3300/0 2024-08-09T14:49:47.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.286+0000 7f0894b40640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0890146f10 addr v1:172.21.15.155:6789/0 2024-08-09T14:49:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.286+0000 7f0894b40640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08901461c0 con 0x7f0890146f10 2024-08-09T14:49:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.286+0000 7f0894b40640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.286+0000 7f0894b40640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f08900b26e0 con 0x7f08900b22e0 2024-08-09T14:49:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.286+0000 7f0894b40640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.286+0000 7f0894b40640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f08900b2860 con 0x7f08900b0160 2024-08-09T14:49:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.286+0000 7f0894b40640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.286+0000 7f0894b40640 10 monclient(hunting): authenticate will time out at 1194.323789s 2024-08-09T14:49:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.287+0000 7f088e59c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08900b0160 0x7f08900b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.287+0000 7f088e59c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08900b0160 0x7f08900b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:35820/0 (socket says 172.21.15.155:35820) 2024-08-09T14:49:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.287+0000 7f088e59c640 1 -- 172.21.15.155:0/2832384304 learned_addr learned my addr 172.21.15.155:0/2832384304 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:49:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.287+0000 7f088e59c640 10 monclient(hunting): get_auth_request con 0x7f08900b0160 auth_method 0 2024-08-09T14:49:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.287+0000 7f088e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:47.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.287+0000 7f088e59c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:47.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.287+0000 7f088e59c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:47.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.288+0000 7f088e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:47.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.288+0000 7f088e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:47.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.288+0000 7f088e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:47.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.290+0000 7f088e59c640 10 monclient(hunting): handle_auth_done global_id 4142 payload 274 2024-08-09T14:49:47.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.290+0000 7f088e59c640 1 -- 172.21.15.155:0/2832384304 >> v2:172.21.15.194:3300/0 conn(0x7f08900b22e0 msgr2=0x7f08901469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:47.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.290+0000 7f088e59c640 1 --2- 172.21.15.155:0/2832384304 >> v2:172.21.15.194:3300/0 conn(0x7f08900b22e0 0x7f08901469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:47.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.290+0000 7f088e59c640 1 -- 172.21.15.155:0/2832384304 >> v1:172.21.15.155:6789/0 conn(0x7f0890146f10 legacy=0x7f08900b3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:47.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.290+0000 7f088e59c640 10 monclient: _finish_hunting 0 2024-08-09T14:49:47.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.290+0000 7f088e59c640 1 monclient: found mon.noname-b 2024-08-09T14:49:47.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.290+0000 7f088e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:47.294 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.290+0000 7f088e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:49:47.294 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.290+0000 7f088e59c640 1 -- 172.21.15.155:0/2832384304 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08900b3f50 con 0x7f08900b0160 2024-08-09T14:49:47.294 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.292+0000 7f088e59c640 1 --2- 172.21.15.155:0/2832384304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08900b0160 0x7f08900b0540 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0880009870 tx=0x7f088002b7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb15bf27d51e1a5e server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:47.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f088dd9b640 1 -- 172.21.15.155:0/2832384304 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0880038070 con 0x7f08900b0160 2024-08-09T14:49:47.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f088dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:47.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f088dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:49:47.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f088dd9b640 10 monclient: dump: 2024-08-09T14:49:47.299 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:49:47.299 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:47.300 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:49:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f088dd9b640 10 monclient: _finish_auth 0 2024-08-09T14:49:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f088dd9b640 10 monclient: _check_auth_tickets 2024-08-09T14:49:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f088dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f088dd9b640 1 -- 172.21.15.155:0/2832384304 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0880033620 con 0x7f08900b0160 2024-08-09T14:49:47.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f088dd9b640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f088dd9b640 1 -- 172.21.15.155:0/2832384304 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08800339f0 con 0x7f08900b0160 2024-08-09T14:49:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f088dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f088dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:49:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f088dd9b640 10 monclient: dump: 2024-08-09T14:49:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:49:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:47.301 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:47.302 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:47.302 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:49:47.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f0894b40640 5 monclient: authenticate success, global_id 4142 2024-08-09T14:49:47.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f0894b40640 10 monclient: get_monmap_and_config success 2024-08-09T14:49:47.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.296+0000 7f0894b40640 4 set_mon_vals no callback set 2024-08-09T14:49:47.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.297+0000 7f0894b40640 10 monclient: shutdown 2024-08-09T14:49:47.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.297+0000 7f0894b40640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:47.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.297+0000 7f0894b40640 1 -- 172.21.15.155:0/2832384304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08900b0160 msgr2=0x7f08900b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:47.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.297+0000 7f0894b40640 1 --2- 172.21.15.155:0/2832384304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08900b0160 0x7f08900b0540 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0880009870 tx=0x7f088002b7d0 comp rx=0 tx=0).stop 2024-08-09T14:49:47.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.297+0000 7f0894b40640 1 -- 172.21.15.155:0/2832384304 shutdown_connections 2024-08-09T14:49:47.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.297+0000 7f0894b40640 1 --2- 172.21.15.155:0/2832384304 >> v2:172.21.15.194:3300/0 conn(0x7f08900b22e0 0x7f08901469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:47.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.297+0000 7f0894b40640 1 --2- 172.21.15.155:0/2832384304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08900b0160 0x7f08900b0540 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:47.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.297+0000 7f0894b40640 1 -- 172.21.15.155:0/2832384304 >> 172.21.15.155:0/2832384304 conn(0x7f089006dbd0 msgr2=0x7f08900676f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:47.303 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.297+0000 7f0894b40640 1 -- 172.21.15.155:0/2832384304 shutdown_connections 2024-08-09T14:49:47.303 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.297+0000 7f0894b40640 1 -- 172.21.15.155:0/2832384304 wait complete. 2024-08-09T14:49:47.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.298+0000 7f0894b40640 10 monclient: build_initial_monmap 2024-08-09T14:49:47.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.299+0000 7f0894b40640 10 monclient: monmap: 2024-08-09T14:49:47.307 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:49:47.307 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:49:47.307 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:49:47.307 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:49:47.307 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:47.307 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:47.307 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:49:47.307 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:49:47.307 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:49:47.308 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:49:47.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.299+0000 7f0894b40640 1 librados: starting msgr at 2024-08-09T14:49:47.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.299+0000 7f0894b40640 1 librados: starting objecter 2024-08-09T14:49:47.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.299+0000 7f0894b40640 1 Processor -- start 2024-08-09T14:49:47.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.299+0000 7f0894b40640 1 -- start start 2024-08-09T14:49:47.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.299+0000 7f0894b40640 1 librados: setting wanted keys 2024-08-09T14:49:47.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.299+0000 7f0894b40640 1 librados: calling monclient init 2024-08-09T14:49:47.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.299+0000 7f0894b40640 10 monclient: init 2024-08-09T14:49:47.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:47.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 10 monclient: _add_conns ranks=[2,1,0] 2024-08-09T14:49:47.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08900b22e0 0x7f08900ab000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:47.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 10 monclient(hunting): picked mon.noname-c con 0x7f08900b22e0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:47.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0890146f10 0x7f08900aa700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:47.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 10 monclient(hunting): picked mon.noname-b con 0x7f0890146f10 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:49:47.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0890149770 0x7f089013fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:47.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 10 monclient(hunting): picked mon.noname-a con 0x7f0890149770 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:47.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:47.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f089014d6d0 con 0x7f0890149770 2024-08-09T14:49:47.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:47.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f089014d550 con 0x7f0890146f10 2024-08-09T14:49:47.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:47.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f089014d850 con 0x7f08900b22e0 2024-08-09T14:49:47.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:47.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f0894b40640 10 monclient(hunting): authenticate will time out at 1194.337526s 2024-08-09T14:49:47.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.300+0000 7f088e59c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08900b22e0 0x7f08900ab000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:47.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.301+0000 7f0887fff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0890146f10 0x7f08900aa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:47.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.301+0000 7f088e59c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08900b22e0 0x7f08900ab000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:35824/0 (socket says 172.21.15.155:35824) 2024-08-09T14:49:47.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.301+0000 7f088e59c640 1 -- 172.21.15.155:0/2346908217 learned_addr learned my addr 172.21.15.155:0/2346908217 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:49:47.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.301+0000 7f088e59c640 10 monclient(hunting): get_auth_request con 0x7f08900b22e0 auth_method 0 2024-08-09T14:49:47.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.301+0000 7f088e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:47.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.301+0000 7f088e59c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:47.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.301+0000 7f088e59c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:47.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.301+0000 7f0887fff640 10 monclient(hunting): get_auth_request con 0x7f0890146f10 auth_method 0 2024-08-09T14:49:47.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.301+0000 7f0887fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:47.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.301+0000 7f0887fff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:47.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.301+0000 7f0887fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:47.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.301+0000 7f088e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:47.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.302+0000 7f088e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:47.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.302+0000 7f088e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:47.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.302+0000 7f0887fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:47.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.302+0000 7f0887fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:47.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.302+0000 7f0887fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:47.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.304+0000 7f088e59c640 10 monclient(hunting): handle_auth_done global_id 4145 payload 824 2024-08-09T14:49:47.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.304+0000 7f088e59c640 1 -- 172.21.15.155:0/2346908217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0890146f10 msgr2=0x7f08900aa700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:47.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.304+0000 7f088e59c640 1 --2- 172.21.15.155:0/2346908217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0890146f10 0x7f08900aa700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:47.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.304+0000 7f088e59c640 1 -- 172.21.15.155:0/2346908217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0890149770 msgr2=0x7f089013fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:47.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.304+0000 7f088e59c640 1 --2- 172.21.15.155:0/2346908217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0890149770 0x7f089013fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:47.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.304+0000 7f088e59c640 10 monclient: _finish_hunting 0 2024-08-09T14:49:47.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.304+0000 7f088e59c640 1 monclient: found mon.noname-c 2024-08-09T14:49:47.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.304+0000 7f088e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:47.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.304+0000 7f088e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:49:47.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.304+0000 7f088e59c640 1 -- 172.21.15.155:0/2346908217 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08900abd90 con 0x7f08900b22e0 2024-08-09T14:49:47.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.307+0000 7f088e59c640 1 --2- 172.21.15.155:0/2346908217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08900b22e0 0x7f08900ab000 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0880009990 tx=0x7f08800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:47.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.307+0000 7f0886ffd640 10 client.?.objecter ms_handle_connect 0x7f08900b22e0 2024-08-09T14:49:47.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.307+0000 7f0886ffd640 10 client.?.objecter resend_mon_ops 2024-08-09T14:49:47.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.312+0000 7f0886ffd640 1 -- 172.21.15.155:0/2346908217 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0880043070 con 0x7f08900b22e0 2024-08-09T14:49:47.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.312+0000 7f0886ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:47.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.312+0000 7f0886ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:49:47.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.312+0000 7f0886ffd640 10 monclient: dump: 2024-08-09T14:49:47.314 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:49:47.314 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:47.314 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:47.314 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:47.314 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:47.314 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:47.314 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:47.315 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:47.315 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:47.315 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:49:47.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.312+0000 7f0886ffd640 10 monclient: _finish_auth 0 2024-08-09T14:49:47.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.312+0000 7f0886ffd640 10 monclient: _check_auth_tickets 2024-08-09T14:49:47.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.312+0000 7f0886ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:47.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.312+0000 7f0886ffd640 1 -- 172.21.15.155:0/2346908217 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08800328d0 con 0x7f08900b22e0 2024-08-09T14:49:47.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.312+0000 7f0886ffd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:47.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.312+0000 7f0886ffd640 1 -- 172.21.15.155:0/2346908217 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0880038040 con 0x7f08900b22e0 2024-08-09T14:49:47.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.312+0000 7f0886ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:47.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.312+0000 7f0886ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:49:47.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.312+0000 7f0886ffd640 10 monclient: dump: 2024-08-09T14:49:47.315 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:49:47.315 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:47.316 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:47.316 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:47.316 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:47.316 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:47.316 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:47.316 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:47.316 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:47.316 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:49:47.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.312+0000 7f088d59a640 4 set_mon_vals no callback set 2024-08-09T14:49:47.317 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.313+0000 7f0894b40640 5 monclient: authenticate success, global_id 4145 2024-08-09T14:49:47.317 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.313+0000 7f0894b40640 10 monclient: _renew_subs 2024-08-09T14:49:47.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.313+0000 7f0894b40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:47.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.313+0000 7f0894b40640 1 -- 172.21.15.155:0/2346908217 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08900ac000 con 0x7f08900b22e0 2024-08-09T14:49:47.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.313+0000 7f0894b40640 10 client.4145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:49:47.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.313+0000 7f0894b40640 10 monclient: _renew_subs 2024-08-09T14:49:47.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.313+0000 7f0894b40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:47.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.313+0000 7f0894b40640 1 -- 172.21.15.155:0/2346908217 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f08900ac5c0 con 0x7f08900b22e0 2024-08-09T14:49:47.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.313+0000 7f0894b40640 1 librados: init done 2024-08-09T14:49:47.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.315+0000 7f0894b40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:47.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.315+0000 7f0894b40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:47.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.315+0000 7f0894b40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:47.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.315+0000 7f0894b40640 1 -- 172.21.15.155:0/2346908217 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08900738a0 con 0x7f08900b22e0 2024-08-09T14:49:47.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.455+0000 7f0886ffd640 1 -- 172.21.15.155:0/2346908217 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0880046070 con 0x7f08900b22e0 2024-08-09T14:49:47.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.456+0000 7f0886ffd640 1 -- 172.21.15.155:0/2346908217 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f088003fb30 con 0x7f08900b22e0 2024-08-09T14:49:47.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.456+0000 7f0886ffd640 10 client.4145.objecter ms_dispatch 0x7f0890000b90 osd_map(10..10 src has 1..10) 2024-08-09T14:49:47.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.456+0000 7f0886ffd640 3 client.4145.objecter handle_osd_map got epochs [10,10] > 0 2024-08-09T14:49:47.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.456+0000 7f0886ffd640 3 client.4145.objecter handle_osd_map decoding full epoch 10 2024-08-09T14:49:47.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.456+0000 7f0886ffd640 20 client.4145.objecter dump_active .. 0 homeless 2024-08-09T14:49:47.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.456+0000 7f0886ffd640 1 -- 172.21.15.155:0/2346908217 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7f08900738a0 con 0x7f08900b22e0 2024-08-09T14:49:47.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.456+0000 7f0886ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:47.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.456+0000 7f0886ffd640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:49:47.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.556+0000 7f0894b40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:49:47.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.556+0000 7f0894b40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:49:47.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.556+0000 7f0894b40640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:47.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.556+0000 7f0894b40640 1 -- 172.21.15.155:0/2346908217 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f08900b1620 con 0x7f08900b22e0 2024-08-09T14:49:47.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.623+0000 7f0886ffd640 1 -- 172.21.15.155:0/2346908217 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mgrmap(e 2) ==== 45274+0+0 (secure 0 0 0) 0x7f088003b040 con 0x7f08900b22e0 2024-08-09T14:49:47.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.789+0000 7f0886ffd640 1 -- 172.21.15.155:0/2346908217 <== mon.2 v2:172.21.15.155:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f08900b1620 con 0x7f08900b22e0 2024-08-09T14:49:47.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.789+0000 7f0886ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:49:47.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.789+0000 7f0886ffd640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:49:47.791 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:49:47.791 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":10,"fsid":"f3d21f4d-627f-4083-bf15-ae5f3fe632de","created":"2024-08-09T14:49:41.321598+0000","modified":"2024-08-09T14:49:46.978521+0000","last_up_change":"0.000000","last_in_change":"2024-08-09T14:49:46.978521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6526d474-661b-48e6-bcc7-6288c7477f78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"87673c5a-b124-419e-98da-17a7b7fbdf25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a14a708a-903f-4de2-8228-79e79ed135e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8e139d50-e25c-4428-9edf-c696189cb40e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0331386-2f50-4bf9-b72e-24c80f9fe9a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0f901e55-2ba8-4144-9802-3a109884254c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4b8215-814e-4b5a-9877-0ab63818b0ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"26ed84b6-639d-4ccd-bb1d-c64b593ada7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-09T14:49:47.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 10 librados: watch_flush enter 2024-08-09T14:49:47.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 10 librados: watch_flush exit 2024-08-09T14:49:47.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 20 client.4145.objecter shutdown clearing up homeless session... 2024-08-09T14:49:47.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 10 client.4145.objecter successfully canceled tick 2024-08-09T14:49:47.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 10 monclient: shutdown 2024-08-09T14:49:47.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:47.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 1 -- 172.21.15.155:0/2346908217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08900b22e0 msgr2=0x7f08900ab000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:47.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 1 --2- 172.21.15.155:0/2346908217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08900b22e0 0x7f08900ab000 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0880009990 tx=0x7f08800046c0 comp rx=0 tx=0).stop 2024-08-09T14:49:47.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 1 -- 172.21.15.155:0/2346908217 shutdown_connections 2024-08-09T14:49:47.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 1 --2- 172.21.15.155:0/2346908217 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0890149770 0x7f089013fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:47.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 1 --2- 172.21.15.155:0/2346908217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0890146f10 0x7f08900aa700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:47.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 1 --2- 172.21.15.155:0/2346908217 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f08900b22e0 0x7f08900ab000 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:47.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 1 -- 172.21.15.155:0/2346908217 >> 172.21.15.155:0/2346908217 conn(0x7f089006dbd0 msgr2=0x7f08900b5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:47.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 1 -- 172.21.15.155:0/2346908217 shutdown_connections 2024-08-09T14:49:47.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.790+0000 7f0894b40640 1 -- 172.21.15.155:0/2346908217 wait complete. 2024-08-09T14:49:47.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:49:47.791+0000 7f0894b40640 1 librados: shutdown 2024-08-09T14:49:47.806 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-08-09T14:49:47.807 INFO:tasks.ceph:Waiting for OSDs to come up 2024-08-09T14:49:48.613 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-09T14:49:48.758 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.757+0000 7f9c94e4d640 10 monclient: get_monmap_and_config 2024-08-09T14:49:48.759 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.757+0000 7f9c94e4d640 10 monclient: build_initial_monmap 2024-08-09T14:49:48.759 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.757+0000 7f9c94e4d640 10 monclient: monmap: 2024-08-09T14:49:48.759 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:49:48.759 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:48.759 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:49:48.758387+0000 2024-08-09T14:49:48.759 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:49:48.758387+0000 2024-08-09T14:49:48.759 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:48.759 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:48.759 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:49:48.759 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:49:48.759 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:49:48.760 INFO:teuthology.misc.health.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:49:48.760 INFO:teuthology.misc.health.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:49:48.760 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:49:48.761 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.758+0000 7f9c94e4d640 1 Processor -- start 2024-08-09T14:49:48.761 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.758+0000 7f9c94e4d640 1 -- start start 2024-08-09T14:49:48.761 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.758+0000 7f9c94e4d640 10 monclient: init 2024-08-09T14:49:48.761 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.759+0000 7f9c94e4d640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:48.761 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.759+0000 7f9c94e4d640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-08-09T14:49:48.761 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.759+0000 7f9c94e4d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c9006d740 addr v1:172.21.15.194:6789/0 2024-08-09T14:49:48.761 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.759+0000 7f9c94e4d640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f9c900b4550 0x7f9c900b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:48.761 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.759+0000 7f9c94e4d640 10 monclient(hunting): picked mon.noname-a con 0x7f9c900b4550 addr v2:172.21.15.155:3300/0 2024-08-09T14:49:48.761 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.759+0000 7f9c94e4d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c90142ce0 0x7f9c90143150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:48.762 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.759+0000 7f9c94e4d640 10 monclient(hunting): picked mon.noname-b con 0x7f9c90142ce0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:48.762 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.759+0000 7f9c94e4d640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c9013cdf0 con 0x7f9c9006d740 2024-08-09T14:49:48.762 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.759+0000 7f9c94e4d640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:48.762 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.759+0000 7f9c94e4d640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f9c9006db20 con 0x7f9c900b4550 2024-08-09T14:49:48.762 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.759+0000 7f9c94e4d640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:48.762 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.759+0000 7f9c94e4d640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f9c900b5180 con 0x7f9c90142ce0 2024-08-09T14:49:48.762 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.759+0000 7f9c94e4d640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:48.762 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.759+0000 7f9c94e4d640 10 monclient(hunting): authenticate will time out at 1195.796584s 2024-08-09T14:49:48.762 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.760+0000 7f9c8ed9d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c90142ce0 0x7f9c90143150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:48.763 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.760+0000 7f9c8ed9d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c90142ce0 0x7f9c90143150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:35938/0 (socket says 172.21.15.155:35938) 2024-08-09T14:49:48.763 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.760+0000 7f9c8ed9d640 1 -- 172.21.15.155:0/1638557594 learned_addr learned my addr 172.21.15.155:0/1638557594 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:49:48.763 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.760+0000 7f9c8ed9d640 10 monclient(hunting): get_auth_request con 0x7f9c90142ce0 auth_method 0 2024-08-09T14:49:48.763 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.760+0000 7f9c8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:48.763 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.760+0000 7f9c8ed9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:48.763 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.760+0000 7f9c8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:48.763 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.761+0000 7f9c8ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:48.763 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.761+0000 7f9c8ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:48.764 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.761+0000 7f9c8ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:48.765 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.763+0000 7f9c8ed9d640 10 monclient(hunting): handle_auth_done global_id 4163 payload 274 2024-08-09T14:49:48.765 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.763+0000 7f9c8ed9d640 1 -- 172.21.15.155:0/1638557594 >> v2:172.21.15.155:3300/0 conn(0x7f9c900b4550 msgr2=0x7f9c900b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:48.765 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.763+0000 7f9c8ed9d640 1 --2- 172.21.15.155:0/1638557594 >> v2:172.21.15.155:3300/0 conn(0x7f9c900b4550 0x7f9c900b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:48.766 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.763+0000 7f9c8ed9d640 1 -- 172.21.15.155:0/1638557594 >> v1:172.21.15.194:6789/0 conn(0x7f9c9006d740 legacy=0x7f9c9013c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:48.766 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.763+0000 7f9c8ed9d640 10 monclient: _finish_hunting 0 2024-08-09T14:49:48.766 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.763+0000 7f9c8ed9d640 1 monclient: found mon.noname-b 2024-08-09T14:49:48.766 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.763+0000 7f9c8ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:48.766 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.763+0000 7f9c8ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:49:48.766 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.763+0000 7f9c8ed9d640 1 -- 172.21.15.155:0/1638557594 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c90143690 con 0x7f9c90142ce0 2024-08-09T14:49:48.767 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.766+0000 7f9c8ed9d640 1 --2- 172.21.15.155:0/1638557594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c90142ce0 0x7f9c90143150 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009e50 tx=0x7f9c8802bf00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42a74ab8df994dcc server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:48.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.772+0000 7f9c8d59a640 1 -- 172.21.15.155:0/1638557594 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8802caa0 con 0x7f9c90142ce0 2024-08-09T14:49:48.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.772+0000 7f9c8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:48.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.772+0000 7f9c8d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:49:48.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.772+0000 7f9c8d59a640 10 monclient: dump: 2024-08-09T14:49:48.776 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:49:48.776 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:48.777 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:48.777 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:48.777 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:48.777 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:48.777 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:48.777 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:48.777 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:48.777 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:49:48.777 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.772+0000 7f9c8d59a640 10 monclient: _finish_auth 0 2024-08-09T14:49:48.777 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.772+0000 7f9c8d59a640 10 monclient: _check_auth_tickets 2024-08-09T14:49:48.777 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.772+0000 7f9c8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:48.777 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.772+0000 7f9c94e4d640 5 monclient: authenticate success, global_id 4163 2024-08-09T14:49:48.777 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.772+0000 7f9c94e4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:49:48.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.772+0000 7f9c8d59a640 1 -- 172.21.15.155:0/1638557594 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c8802cc40 con 0x7f9c90142ce0 2024-08-09T14:49:48.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.772+0000 7f9c8d59a640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:48.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.772+0000 7f9c94e4d640 10 monclient: get_monmap_and_config success 2024-08-09T14:49:48.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.772+0000 7f9c94e4d640 4 set_mon_vals no callback set 2024-08-09T14:49:48.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.773+0000 7f9c8d59a640 1 -- 172.21.15.155:0/1638557594 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88034610 con 0x7f9c90142ce0 2024-08-09T14:49:48.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.773+0000 7f9c8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:48.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.773+0000 7f9c8d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:49:48.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.773+0000 7f9c8d59a640 10 monclient: dump: 2024-08-09T14:49:48.778 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:49:48.778 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:48.778 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:48.778 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:48.778 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:48.779 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:48.779 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:48.779 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:48.779 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:48.779 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:49:48.779 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.773+0000 7f9c94e4d640 10 monclient: shutdown 2024-08-09T14:49:48.779 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.773+0000 7f9c94e4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:48.779 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.773+0000 7f9c94e4d640 1 -- 172.21.15.155:0/1638557594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c90142ce0 msgr2=0x7f9c90143150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:48.779 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.773+0000 7f9c94e4d640 1 --2- 172.21.15.155:0/1638557594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c90142ce0 0x7f9c90143150 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009e50 tx=0x7f9c8802bf00 comp rx=0 tx=0).stop 2024-08-09T14:49:48.779 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.774+0000 7f9c94e4d640 1 -- 172.21.15.155:0/1638557594 shutdown_connections 2024-08-09T14:49:48.779 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.774+0000 7f9c94e4d640 1 --2- 172.21.15.155:0/1638557594 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c90142ce0 0x7f9c90143150 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:48.779 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.774+0000 7f9c94e4d640 1 --2- 172.21.15.155:0/1638557594 >> v2:172.21.15.155:3300/0 conn(0x7f9c900b4550 0x7f9c900b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:48.779 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.774+0000 7f9c94e4d640 1 -- 172.21.15.155:0/1638557594 >> 172.21.15.155:0/1638557594 conn(0x7f9c900b8d30 msgr2=0x7f9c900b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:48.779 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.774+0000 7f9c94e4d640 1 -- 172.21.15.155:0/1638557594 shutdown_connections 2024-08-09T14:49:48.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.775+0000 7f9c94e4d640 1 -- 172.21.15.155:0/1638557594 wait complete. 2024-08-09T14:49:48.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.776+0000 7f9c94e4d640 10 monclient: build_initial_monmap 2024-08-09T14:49:48.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.776+0000 7f9c94e4d640 10 monclient: monmap: 2024-08-09T14:49:48.783 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:49:48.783 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:49:48.783 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-08-09T14:49:48.783 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-08-09T14:49:48.783 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:48.783 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:48.784 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:49:48.784 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:49:48.784 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:49:48.784 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:49:48.784 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.776+0000 7f9c94e4d640 1 librados: starting msgr at 2024-08-09T14:49:48.784 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.776+0000 7f9c94e4d640 1 librados: starting objecter 2024-08-09T14:49:48.784 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.776+0000 7f9c94e4d640 1 Processor -- start 2024-08-09T14:49:48.784 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.776+0000 7f9c94e4d640 1 -- start start 2024-08-09T14:49:48.784 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.776+0000 7f9c94e4d640 1 librados: setting wanted keys 2024-08-09T14:49:48.784 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.776+0000 7f9c94e4d640 1 librados: calling monclient init 2024-08-09T14:49:48.784 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.776+0000 7f9c94e4d640 10 monclient: init 2024-08-09T14:49:48.784 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:48.784 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 10 monclient: _add_conns ranks=[1,0,2] 2024-08-09T14:49:48.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c9006d740 0x7f9c90146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:48.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 10 monclient(hunting): picked mon.noname-b con 0x7f9c9006d740 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:49:48.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c900b4550 0x7f9c90136c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:48.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 10 monclient(hunting): picked mon.noname-a con 0x7f9c900b4550 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:48.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c90142ce0 0x7f9c90143f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:48.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 10 monclient(hunting): picked mon.noname-c con 0x7f9c90142ce0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:48.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:48.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9013cfc0 con 0x7f9c900b4550 2024-08-09T14:49:48.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:48.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9c901439e0 con 0x7f9c9006d740 2024-08-09T14:49:48.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:48.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f9c9014d650 con 0x7f9c90142ce0 2024-08-09T14:49:48.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:48.786 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.777+0000 7f9c94e4d640 10 monclient(hunting): authenticate will time out at 1195.814934s 2024-08-09T14:49:48.786 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.778+0000 7f9c8ed9d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c90142ce0 0x7f9c90143f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:48.786 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.778+0000 7f9c8e59c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c9006d740 0x7f9c90146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:48.786 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.779+0000 7f9c8ed9d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c90142ce0 0x7f9c90143f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:35954/0 (socket says 172.21.15.155:35954) 2024-08-09T14:49:48.786 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.779+0000 7f9c8ed9d640 1 -- 172.21.15.155:0/3039566859 learned_addr learned my addr 172.21.15.155:0/3039566859 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:49:48.786 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.779+0000 7f9c8ed9d640 10 monclient(hunting): get_auth_request con 0x7f9c90142ce0 auth_method 0 2024-08-09T14:49:48.786 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.779+0000 7f9c8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:48.786 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.779+0000 7f9c8ed9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:48.786 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.779+0000 7f9c8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:48.786 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.779+0000 7f9c8e59c640 10 monclient(hunting): get_auth_request con 0x7f9c9006d740 auth_method 0 2024-08-09T14:49:48.786 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.779+0000 7f9c8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:48.786 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.779+0000 7f9c8e59c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:48.786 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.779+0000 7f9c8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:48.787 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.779+0000 7f9c8ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:48.787 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.779+0000 7f9c8ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:48.787 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.779+0000 7f9c8ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:48.787 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.780+0000 7f9c8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:48.787 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.780+0000 7f9c8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:48.787 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.780+0000 7f9c8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:48.789 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.783+0000 7f9c8ed9d640 10 monclient(hunting): handle_auth_done global_id 4166 payload 824 2024-08-09T14:49:48.789 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.783+0000 7f9c8ed9d640 1 -- 172.21.15.155:0/3039566859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c9006d740 msgr2=0x7f9c90146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:48.789 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.783+0000 7f9c8ed9d640 1 --2- 172.21.15.155:0/3039566859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c9006d740 0x7f9c90146650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:48.789 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.784+0000 7f9c8ed9d640 1 -- 172.21.15.155:0/3039566859 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c900b4550 msgr2=0x7f9c90136c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:48.789 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.784+0000 7f9c8ed9d640 1 --2- 172.21.15.155:0/3039566859 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c900b4550 0x7f9c90136c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:48.789 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.784+0000 7f9c8ed9d640 10 monclient: _finish_hunting 0 2024-08-09T14:49:48.789 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.784+0000 7f9c8ed9d640 1 monclient: found mon.noname-c 2024-08-09T14:49:48.789 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.784+0000 7f9c8ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:48.790 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.784+0000 7f9c8ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:49:48.790 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.784+0000 7f9c8ed9d640 1 -- 172.21.15.155:0/3039566859 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c90145be0 con 0x7f9c90142ce0 2024-08-09T14:49:48.790 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.787+0000 7f9c8ed9d640 1 --2- 172.21.15.155:0/3039566859 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c90142ce0 0x7f9c90143f40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009e50 tx=0x7f9c88004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:48.790 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.787+0000 7f9c7effd640 10 client.?.objecter ms_handle_connect 0x7f9c90142ce0 2024-08-09T14:49:48.790 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.787+0000 7f9c7effd640 10 client.?.objecter resend_mon_ops 2024-08-09T14:49:48.796 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c7effd640 1 -- 172.21.15.155:0/3039566859 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c88043020 con 0x7f9c90142ce0 2024-08-09T14:49:48.796 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:48.796 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c7effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:49:48.796 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c7effd640 10 monclient: dump: 2024-08-09T14:49:48.796 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:49:48.796 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:48.796 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:48.796 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:48.797 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:48.797 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:48.797 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:48.797 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:48.797 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:48.797 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:49:48.797 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c7effd640 10 monclient: _finish_auth 0 2024-08-09T14:49:48.797 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c7effd640 10 monclient: _check_auth_tickets 2024-08-09T14:49:48.797 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:48.797 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c7effd640 1 -- 172.21.15.155:0/3039566859 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c88007e50 con 0x7f9c90142ce0 2024-08-09T14:49:48.797 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c7effd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:48.797 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c7effd640 1 -- 172.21.15.155:0/3039566859 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8803e690 con 0x7f9c90142ce0 2024-08-09T14:49:48.797 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:48.798 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:49:48.798 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c7effd640 10 monclient: dump: 2024-08-09T14:49:48.798 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:49:48.798 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:48.798 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:48.798 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:48.798 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:48.798 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:48.798 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:48.798 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:48.798 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:48.798 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:49:48.798 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c8cd99640 4 set_mon_vals no callback set 2024-08-09T14:49:48.799 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c94e4d640 5 monclient: authenticate success, global_id 4166 2024-08-09T14:49:48.799 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c94e4d640 10 monclient: _renew_subs 2024-08-09T14:49:48.799 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c94e4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:48.799 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.793+0000 7f9c94e4d640 1 -- 172.21.15.155:0/3039566859 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c90145ec0 con 0x7f9c90142ce0 2024-08-09T14:49:48.799 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.794+0000 7f9c94e4d640 10 client.4166.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:49:48.799 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.794+0000 7f9c94e4d640 10 monclient: _renew_subs 2024-08-09T14:49:48.799 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.794+0000 7f9c94e4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:48.799 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.794+0000 7f9c94e4d640 1 -- 172.21.15.155:0/3039566859 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c9013fc10 con 0x7f9c90142ce0 2024-08-09T14:49:48.799 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.794+0000 7f9c94e4d640 1 librados: init done 2024-08-09T14:49:48.800 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.795+0000 7f9c94e4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:48.800 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.795+0000 7f9c94e4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:48.800 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.795+0000 7f9c94e4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:48.800 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.795+0000 7f9c94e4d640 1 -- 172.21.15.155:0/3039566859 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c900b49a0 con 0x7f9c90142ce0 2024-08-09T14:49:48.831 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.830+0000 7f9c7effd640 1 -- 172.21.15.155:0/3039566859 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 3) ==== 90209+0+0 (secure 0 0 0) 0x7f9c88046020 con 0x7f9c90142ce0 2024-08-09T14:49:48.931 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.930+0000 7f9c7effd640 1 -- 172.21.15.155:0/3039566859 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f9c88079a20 con 0x7f9c90142ce0 2024-08-09T14:49:48.931 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.930+0000 7f9c7effd640 10 client.4166.objecter ms_dispatch 0x7f9c90000b90 osd_map(10..10 src has 1..10) 2024-08-09T14:49:48.931 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.930+0000 7f9c7effd640 3 client.4166.objecter handle_osd_map got epochs [10,10] > 0 2024-08-09T14:49:48.931 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.930+0000 7f9c7effd640 3 client.4166.objecter handle_osd_map decoding full epoch 10 2024-08-09T14:49:48.931 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:48.930+0000 7f9c7effd640 20 client.4166.objecter dump_active .. 0 homeless 2024-08-09T14:49:49.106 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.105+0000 7f9c7effd640 1 -- 172.21.15.155:0/3039566859 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73482 (secure 0 0 0) 0x7f9c8804d030 con 0x7f9c90142ce0 2024-08-09T14:49:49.106 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.105+0000 7f9c7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:49.106 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.105+0000 7f9c7effd640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:49:49.176 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.174+0000 7f9c94e4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:49:49.176 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.174+0000 7f9c94e4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:49:49.176 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.174+0000 7f9c94e4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:49:49.176 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.174+0000 7f9c94e4d640 1 -- 172.21.15.155:0/3039566859 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9c9006e840 con 0x7f9c90142ce0 2024-08-09T14:49:49.248 INFO:teuthology.misc.health.smithi155.stdout: 2024-08-09T14:49:49.248 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"f3d21f4d-627f-4083-bf15-ae5f3fe632de","created":"2024-08-09T14:49:41.321598+0000","modified":"2024-08-09T14:49:46.978521+0000","last_up_change":"0.000000","last_in_change":"2024-08-09T14:49:46.978521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6526d474-661b-48e6-bcc7-6288c7477f78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"87673c5a-b124-419e-98da-17a7b7fbdf25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a14a708a-903f-4de2-8228-79e79ed135e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8e139d50-e25c-4428-9edf-c696189cb40e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0331386-2f50-4bf9-b72e-24c80f9fe9a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0f901e55-2ba8-4144-9802-3a109884254c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4b8215-814e-4b5a-9877-0ab63818b0ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"26ed84b6-639d-4ccd-bb1d-c64b593ada7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-09T14:49:49.249 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.245+0000 7f9c7effd640 1 -- 172.21.15.155:0/3039566859 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f9c90145be0 con 0x7f9c90142ce0 2024-08-09T14:49:49.249 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.245+0000 7f9c7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:49:49.249 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.245+0000 7f9c7effd640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:49:49.249 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 10 librados: watch_flush enter 2024-08-09T14:49:49.249 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 10 librados: watch_flush exit 2024-08-09T14:49:49.250 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 20 client.4166.objecter shutdown clearing up homeless session... 2024-08-09T14:49:49.250 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 10 client.4166.objecter successfully canceled tick 2024-08-09T14:49:49.250 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 10 monclient: shutdown 2024-08-09T14:49:49.250 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:49.250 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 1 -- 172.21.15.155:0/3039566859 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c90142ce0 msgr2=0x7f9c90143f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:49.250 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 1 --2- 172.21.15.155:0/3039566859 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c90142ce0 0x7f9c90143f40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f9c88009e50 tx=0x7f9c88004a40 comp rx=0 tx=0).stop 2024-08-09T14:49:49.250 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 1 -- 172.21.15.155:0/3039566859 shutdown_connections 2024-08-09T14:49:49.250 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 1 --2- 172.21.15.155:0/3039566859 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9c90142ce0 0x7f9c90143f40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:49.250 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 1 --2- 172.21.15.155:0/3039566859 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9c900b4550 0x7f9c90136c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:49.250 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 1 --2- 172.21.15.155:0/3039566859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9c9006d740 0x7f9c90146650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:49.250 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 1 -- 172.21.15.155:0/3039566859 >> 172.21.15.155:0/3039566859 conn(0x7f9c900b8d30 msgr2=0x7f9c900b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:49.250 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 1 -- 172.21.15.155:0/3039566859 shutdown_connections 2024-08-09T14:49:49.251 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 1 -- 172.21.15.155:0/3039566859 wait complete. 2024-08-09T14:49:49.251 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:49.247+0000 7f9c94e4d640 1 librados: shutdown 2024-08-09T14:49:49.261 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-08-09T14:49:49.926 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:49.924+0000 7f2c2bb36640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-08-09T14:49:49.998 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T14:49:49.997+0000 7f2c2bb36640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-08-09T14:49:54.703 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:07.486 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:54.761 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:07.524 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:54.772 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:07.555 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:54.826 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:07.589 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:54.873 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:07.655 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:54.911 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:07.693 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:54.933 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:07.714 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:54.938 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:07.714 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:54.972 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:07.753 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:55.004 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:07.780 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:55.145 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:07.910 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:55.206 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:07.971 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:55.337 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:49:55.328+0000 5dce500 -1 Falling back to public interface 2024-08-09T14:49:55.395 INFO:tasks.ceph.osd.6.smithi194.stderr:2024-08-09T14:49:55.388+0000 5dce500 -1 Falling back to public interface 2024-08-09T14:49:55.516 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:49:55.509+0000 5dce500 -1 Falling back to public interface 2024-08-09T14:49:55.530 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:49:55.521+0000 5dce500 -1 Falling back to public interface 2024-08-09T14:49:55.553 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:49:55.545+0000 5dce500 -1 Falling back to public interface 2024-08-09T14:49:55.560 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:08.348 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:55.592 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:08.365 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:55.620 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:08.407 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:55.654 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:08.426 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:55.791 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:49:55.784+0000 5dce500 -1 Falling back to public interface 2024-08-09T14:49:55.842 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:08.623 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:55.868 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:08.630 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:55.972 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:08.753 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:55.986 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:08.768 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.041 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:08.816 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.067 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-09T14:49:56.160 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.158+0000 7fda6c31a640 10 monclient: get_monmap_and_config 2024-08-09T14:49:56.160 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.158+0000 7fda6c31a640 10 monclient: build_initial_monmap 2024-08-09T14:49:56.160 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.158+0000 7fda6c31a640 10 monclient: monmap: 2024-08-09T14:49:56.160 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:49:56.160 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:56.160 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:49:56.159797+0000 2024-08-09T14:49:56.160 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:49:56.159797+0000 2024-08-09T14:49:56.160 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:56.160 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:56.160 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:49:56.161 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:49:56.161 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:49:56.161 INFO:teuthology.misc.health.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:49:56.161 INFO:teuthology.misc.health.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:49:56.161 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:49:56.162 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.159+0000 7fda6c31a640 1 Processor -- start 2024-08-09T14:49:56.163 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.159+0000 7fda6c31a640 1 -- start start 2024-08-09T14:49:56.163 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.159+0000 7fda6c31a640 10 monclient: init 2024-08-09T14:49:56.163 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda6c31a640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:56.163 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda6c31a640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-08-09T14:49:56.163 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda6c31a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda6406d740 addr v1:172.21.15.155:6789/0 2024-08-09T14:49:56.163 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda6c31a640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fda640b4550 0x7fda640b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:56.163 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda6c31a640 10 monclient(hunting): picked mon.noname-c con 0x7fda640b4550 addr v2:172.21.15.194:3300/0 2024-08-09T14:49:56.163 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda6c31a640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda64142ce0 0x7fda64143150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:56.163 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda6c31a640 10 monclient(hunting): picked mon.noname-b con 0x7fda64142ce0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:56.163 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda6c31a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda6413cdf0 con 0x7fda6406d740 2024-08-09T14:49:56.163 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda6c31a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:56.163 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda6c31a640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7fda6406db20 con 0x7fda640b4550 2024-08-09T14:49:56.163 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda6c31a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda6c31a640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fda640b5180 con 0x7fda64142ce0 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda6c31a640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda6c31a640 10 monclient(hunting): authenticate will time out at 1203.197360s 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.160+0000 7fda698b5640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fda640b4550 0x7fda640b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.161+0000 7fda6a8b7640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda64142ce0 0x7fda64143150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.161+0000 7fda6a0b6640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7fda6406d740 0x7fda6413c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:59712/0 (socket says 172.21.15.155:59712) 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.161+0000 7fda698b5640 10 monclient(hunting): get_auth_request con 0x7fda640b4550 auth_method 0 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.161+0000 7fda6a0b6640 1 -- 172.21.15.155:0/3080622392 learned_addr learned my addr 172.21.15.155:0/3080622392 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.161+0000 7fda698b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.161+0000 7fda698b5640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.161+0000 7fda698b5640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.161+0000 7fda6a8b7640 10 monclient(hunting): get_auth_request con 0x7fda64142ce0 auth_method 0 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.161+0000 7fda6a8b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.161+0000 7fda6a8b7640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:56.164 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.161+0000 7fda6a8b7640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:56.165 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.161+0000 7fda698b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:56.165 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.161+0000 7fda698b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:56.165 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.161+0000 7fda698b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:56.165 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.162+0000 7fda6a8b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:56.166 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.162+0000 7fda6a8b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:56.166 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.162+0000 7fda6a8b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:56.166 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.163+0000 7fda698b5640 10 monclient(hunting): handle_auth_done global_id 4177 payload 274 2024-08-09T14:49:56.166 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.163+0000 7fda698b5640 1 -- 172.21.15.155:0/3080622392 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda64142ce0 msgr2=0x7fda64143150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:49:56.166 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.163+0000 7fda698b5640 1 --2- 172.21.15.155:0/3080622392 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda64142ce0 0x7fda64143150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:56.166 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.163+0000 7fda698b5640 1 -- 172.21.15.155:0/3080622392 >> v1:172.21.15.155:6789/0 conn(0x7fda6406d740 legacy=0x7fda6413c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:56.166 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.163+0000 7fda698b5640 10 monclient: _finish_hunting 0 2024-08-09T14:49:56.166 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.163+0000 7fda698b5640 1 monclient: found mon.noname-c 2024-08-09T14:49:56.166 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.163+0000 7fda698b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:56.166 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.163+0000 7fda698b5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:49:56.166 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.163+0000 7fda698b5640 1 -- 172.21.15.155:0/3080622392 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda64143690 con 0x7fda640b4550 2024-08-09T14:49:56.167 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.165+0000 7fda698b5640 1 --2- 172.21.15.155:0/3080622392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda640b4550 0x7fda640b49a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fda540097a0 tx=0x7fda5402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a9524dd60fb53a1 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:56.171 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda690b4640 1 -- 172.21.15.155:0/3080622392 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54031780 con 0x7fda640b4550 2024-08-09T14:49:56.171 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda690b4640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:56.171 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda690b4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:49:56.171 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda690b4640 10 monclient: dump: 2024-08-09T14:49:56.172 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:49:56.172 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:56.172 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:56.172 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:56.172 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:56.172 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:56.172 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:56.172 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:56.172 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:56.172 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:49:56.172 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda690b4640 10 monclient: _finish_auth 0 2024-08-09T14:49:56.172 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda690b4640 10 monclient: _check_auth_tickets 2024-08-09T14:49:56.172 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda690b4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:56.173 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda6c31a640 5 monclient: authenticate success, global_id 4177 2024-08-09T14:49:56.173 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda690b4640 1 -- 172.21.15.155:0/3080622392 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda54037070 con 0x7fda640b4550 2024-08-09T14:49:56.173 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda6c31a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:49:56.173 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda690b4640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:56.173 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda6c31a640 10 monclient: get_monmap_and_config success 2024-08-09T14:49:56.173 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda6c31a640 4 set_mon_vals no callback set 2024-08-09T14:49:56.173 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda690b4640 1 -- 172.21.15.155:0/3080622392 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54031b40 con 0x7fda640b4550 2024-08-09T14:49:56.173 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda690b4640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:56.173 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda690b4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:49:56.173 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda690b4640 10 monclient: dump: 2024-08-09T14:49:56.173 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:49:56.173 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:56.174 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:56.174 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:56.174 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:56.174 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:56.174 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:56.174 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:56.174 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:56.174 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:49:56.177 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda6c31a640 10 monclient: shutdown 2024-08-09T14:49:56.177 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda6c31a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:56.178 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda6c31a640 1 -- 172.21.15.155:0/3080622392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda640b4550 msgr2=0x7fda640b49a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:56.178 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.170+0000 7fda6c31a640 1 --2- 172.21.15.155:0/3080622392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda640b4550 0x7fda640b49a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fda540097a0 tx=0x7fda5402b820 comp rx=0 tx=0).stop 2024-08-09T14:49:56.178 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.171+0000 7fda6c31a640 1 -- 172.21.15.155:0/3080622392 shutdown_connections 2024-08-09T14:49:56.178 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.171+0000 7fda6c31a640 1 --2- 172.21.15.155:0/3080622392 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda64142ce0 0x7fda64143150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:56.178 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.171+0000 7fda6c31a640 1 --2- 172.21.15.155:0/3080622392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda640b4550 0x7fda640b49a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:56.178 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.171+0000 7fda6c31a640 1 -- 172.21.15.155:0/3080622392 >> 172.21.15.155:0/3080622392 conn(0x7fda640b8d30 msgr2=0x7fda640b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:56.178 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.171+0000 7fda6c31a640 1 -- 172.21.15.155:0/3080622392 shutdown_connections 2024-08-09T14:49:56.178 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.171+0000 7fda6c31a640 1 -- 172.21.15.155:0/3080622392 wait complete. 2024-08-09T14:49:56.178 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.171+0000 7fda6c31a640 10 monclient: build_initial_monmap 2024-08-09T14:49:56.178 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.171+0000 7fda6c31a640 10 monclient: monmap: 2024-08-09T14:49:56.178 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:49:56.178 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:49:56.179 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-08-09T14:49:56.179 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-08-09T14:49:56.179 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:49:56.179 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:56.179 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:49:56.179 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:49:56.179 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:49:56.179 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:49:56.179 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 1 librados: starting msgr at 2024-08-09T14:49:56.179 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 1 librados: starting objecter 2024-08-09T14:49:56.179 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 1 Processor -- start 2024-08-09T14:49:56.179 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 1 -- start start 2024-08-09T14:49:56.179 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 1 librados: setting wanted keys 2024-08-09T14:49:56.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 1 librados: calling monclient init 2024-08-09T14:49:56.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 10 monclient: init 2024-08-09T14:49:56.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 10 monclient: _reopen_session rank -1 2024-08-09T14:49:56.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 10 monclient: _add_conns ranks=[2,1,0] 2024-08-09T14:49:56.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda6406d740 0x7fda64146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:56.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 10 monclient(hunting): picked mon.noname-c con 0x7fda6406d740 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:49:56.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda640b4550 0x7fda64143eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:56.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 10 monclient(hunting): picked mon.noname-b con 0x7fda640b4550 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:49:56.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda64142ce0 0x7fda641443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:56.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 10 monclient(hunting): picked mon.noname-a con 0x7fda64142ce0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:49:56.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:56.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fda6414d650 con 0x7fda64142ce0 2024-08-09T14:49:56.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:56.181 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fda6413cfc0 con 0x7fda640b4550 2024-08-09T14:49:56.181 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:49:56.181 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fda6414d7d0 con 0x7fda6406d740 2024-08-09T14:49:56.181 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 10 monclient(hunting): _renew_subs 2024-08-09T14:49:56.181 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.172+0000 7fda6c31a640 10 monclient(hunting): authenticate will time out at 1203.209881s 2024-08-09T14:49:56.181 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda698b5640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda640b4550 0x7fda64143eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:56.181 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda6a8b7640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda64142ce0 0x7fda641443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:56.181 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda698b5640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda640b4550 0x7fda64143eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:60780/0 (socket says 172.21.15.155:60780) 2024-08-09T14:49:56.181 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda698b5640 1 -- 172.21.15.155:0/1625076925 learned_addr learned my addr 172.21.15.155:0/1625076925 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:49:56.181 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda698b5640 10 monclient(hunting): get_auth_request con 0x7fda640b4550 auth_method 0 2024-08-09T14:49:56.181 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda698b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:56.181 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda698b5640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:56.182 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda698b5640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:56.182 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda6a0b6640 1 --2- 172.21.15.155:0/1625076925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda6406d740 0x7fda64146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:56.182 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda6a8b7640 10 monclient(hunting): get_auth_request con 0x7fda64142ce0 auth_method 0 2024-08-09T14:49:56.182 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda6a8b7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:56.182 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda6a8b7640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:56.182 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda6a8b7640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:56.182 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda698b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:56.182 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda698b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:56.182 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.173+0000 7fda698b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:56.182 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.174+0000 7fda6a0b6640 10 monclient(hunting): get_auth_request con 0x7fda6406d740 auth_method 0 2024-08-09T14:49:56.182 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.174+0000 7fda6a0b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:49:56.183 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.174+0000 7fda6a0b6640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:49:56.183 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.174+0000 7fda6a0b6640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:49:56.183 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.174+0000 7fda6a8b7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:56.183 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.174+0000 7fda6a8b7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:56.183 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.174+0000 7fda6a8b7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:56.183 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.174+0000 7fda6a0b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:49:56.183 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.174+0000 7fda6a0b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:49:56.183 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.174+0000 7fda6a0b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:49:56.183 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.176+0000 7fda698b5640 10 monclient(hunting): handle_auth_done global_id 4180 payload 824 2024-08-09T14:49:56.183 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.176+0000 7fda698b5640 1 -- 172.21.15.155:0/1625076925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda6406d740 msgr2=0x7fda64146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:56.183 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.176+0000 7fda698b5640 1 --2- 172.21.15.155:0/1625076925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda6406d740 0x7fda64146650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:56.183 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.176+0000 7fda698b5640 1 -- 172.21.15.155:0/1625076925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda64142ce0 msgr2=0x7fda641443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:56.184 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.176+0000 7fda698b5640 1 --2- 172.21.15.155:0/1625076925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda64142ce0 0x7fda641443f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:56.184 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.176+0000 7fda698b5640 10 monclient: _finish_hunting 0 2024-08-09T14:49:56.184 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.176+0000 7fda698b5640 1 monclient: found mon.noname-b 2024-08-09T14:49:56.184 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.176+0000 7fda698b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:49:56.184 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.176+0000 7fda698b5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:49:56.184 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.176+0000 7fda698b5640 1 -- 172.21.15.155:0/1625076925 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda64146f90 con 0x7fda640b4550 2024-08-09T14:49:56.185 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.178+0000 7fda698b5640 1 --2- 172.21.15.155:0/1625076925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda640b4550 0x7fda64143eb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fda54034040 tx=0x7fda540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:56.185 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.178+0000 7fda5affd640 10 client.?.objecter ms_handle_connect 0x7fda640b4550 2024-08-09T14:49:56.186 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.178+0000 7fda5affd640 10 client.?.objecter resend_mon_ops 2024-08-09T14:49:56.186 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda5affd640 1 -- 172.21.15.155:0/1625076925 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54032610 con 0x7fda640b4550 2024-08-09T14:49:56.186 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:56.186 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda5affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:49:56.186 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda5affd640 10 monclient: dump: 2024-08-09T14:49:56.186 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:49:56.186 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:56.186 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:56.186 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:56.186 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:56.186 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:56.186 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:56.186 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:56.187 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:56.187 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:49:56.187 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda5affd640 10 monclient: _finish_auth 0 2024-08-09T14:49:56.187 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda5affd640 10 monclient: _check_auth_tickets 2024-08-09T14:49:56.187 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:49:56.187 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda6c31a640 5 monclient: authenticate success, global_id 4180 2024-08-09T14:49:56.187 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda6c31a640 10 monclient: _renew_subs 2024-08-09T14:49:56.187 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda6c31a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:56.187 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda6c31a640 1 -- 172.21.15.155:0/1625076925 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda64145be0 con 0x7fda640b4550 2024-08-09T14:49:56.187 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda6c31a640 10 client.4180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:49:56.187 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda6c31a640 10 monclient: _renew_subs 2024-08-09T14:49:56.187 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda6c31a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:56.187 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda6c31a640 1 -- 172.21.15.155:0/1625076925 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda64146b90 con 0x7fda640b4550 2024-08-09T14:49:56.187 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.181+0000 7fda6c31a640 1 librados: init done 2024-08-09T14:49:56.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.182+0000 7fda6c31a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:56.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.182+0000 7fda6c31a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:56.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.182+0000 7fda6c31a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:56.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.182+0000 7fda6c31a640 1 -- 172.21.15.155:0/1625076925 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda64145fd0 con 0x7fda640b4550 2024-08-09T14:49:56.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.183+0000 7fda5affd640 1 -- 172.21.15.155:0/1625076925 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda54037020 con 0x7fda640b4550 2024-08-09T14:49:56.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.183+0000 7fda5affd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:49:56.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.183+0000 7fda5affd640 1 -- 172.21.15.155:0/1625076925 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54005b90 con 0x7fda640b4550 2024-08-09T14:49:56.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.183+0000 7fda5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:49:56.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.183+0000 7fda5affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:49:56.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.183+0000 7fda5affd640 10 monclient: dump: 2024-08-09T14:49:56.188 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:49:56.188 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:49:56.188 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:56.188 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:49:56.189 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:49:56.189 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:49:56.189 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:49:56.189 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:49:56.189 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:49:56.189 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:49:56.189 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.183+0000 7fda688b3640 4 set_mon_vals no callback set 2024-08-09T14:49:56.205 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:49:56.196+0000 5dce500 -1 Falling back to public interface 2024-08-09T14:49:56.212 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:49:56.204+0000 5dce500 -1 Falling back to public interface 2024-08-09T14:49:56.235 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:09.018 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.239 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:09.004 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.246 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:09.009 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.342 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:09.124 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.361 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:09.143 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.390 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:09.155 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.407 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:09.183 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.460 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.458+0000 7fda5affd640 1 -- 172.21.15.155:0/1625076925 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7fda540327b0 con 0x7fda640b4550 2024-08-09T14:49:56.460 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.459+0000 7fda5affd640 1 --2- 172.21.15.155:0/1625076925 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fda4806d730 0x7fda4806fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:49:56.461 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.459+0000 7fda6a0b6640 1 --2- 172.21.15.155:0/1625076925 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fda4806d730 0x7fda4806fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:49:56.461 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.459+0000 7fda6a0b6640 10 monclient: get_auth_request con 0x7fda4806d730 auth_method 0 2024-08-09T14:49:56.461 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.460+0000 7fda5affd640 10 client.4180.objecter ms_handle_connect 0x7fda4806d730 2024-08-09T14:49:56.461 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.460+0000 7fda6a0b6640 1 --2- 172.21.15.155:0/1625076925 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fda4806d730 0x7fda4806fbf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fda60007fd0 tx=0x7fda60007e80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:49:56.464 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.462+0000 7fda5affd640 1 -- 172.21.15.155:0/1625076925 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fda540ab010 con 0x7fda640b4550 2024-08-09T14:49:56.464 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.462+0000 7fda5affd640 10 client.4180.objecter ms_dispatch 0x7fda64000b90 osd_map(10..10 src has 1..10) 2024-08-09T14:49:56.464 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.462+0000 7fda5affd640 3 client.4180.objecter handle_osd_map got epochs [10,10] > 0 2024-08-09T14:49:56.464 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.462+0000 7fda5affd640 3 client.4180.objecter handle_osd_map decoding full epoch 10 2024-08-09T14:49:56.464 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.462+0000 7fda5affd640 20 client.4180.objecter dump_active .. 0 homeless 2024-08-09T14:49:56.474 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.472+0000 7fda5affd640 1 -- 172.21.15.155:0/1625076925 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7fda64146f90 con 0x7fda640b4550 2024-08-09T14:49:56.474 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.472+0000 7fda5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:49:56.474 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.472+0000 7fda5affd640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:49:56.560 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:09.342 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.567 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:09.330 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.664 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:09.451 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.673 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:09.455 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.685 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.683+0000 7fda6c31a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:49:56.685 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.683+0000 7fda6c31a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:49:56.685 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.683+0000 7fda6c31a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:49:56.685 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.683+0000 7fda6c31a640 1 -- 172.21.15.155:0/1625076925 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda64145db0 con 0x7fda640b4550 2024-08-09T14:49:56.710 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:09.475 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.716 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:09.497 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.729 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.728+0000 7fda5affd640 1 -- 172.21.15.155:0/1625076925 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fda54040070 con 0x7fda640b4550 2024-08-09T14:49:56.729 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.728+0000 7fda5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:49:56.729 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.728+0000 7fda5affd640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:49:56.729 INFO:teuthology.misc.health.smithi155.stdout: 2024-08-09T14:49:56.729 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"f3d21f4d-627f-4083-bf15-ae5f3fe632de","created":"2024-08-09T14:49:41.321598+0000","modified":"2024-08-09T14:49:46.978521+0000","last_up_change":"0.000000","last_in_change":"2024-08-09T14:49:46.978521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6526d474-661b-48e6-bcc7-6288c7477f78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"87673c5a-b124-419e-98da-17a7b7fbdf25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a14a708a-903f-4de2-8228-79e79ed135e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8e139d50-e25c-4428-9edf-c696189cb40e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0331386-2f50-4bf9-b72e-24c80f9fe9a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0f901e55-2ba8-4144-9802-3a109884254c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4b8215-814e-4b5a-9877-0ab63818b0ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"26ed84b6-639d-4ccd-bb1d-c64b593ada7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-09T14:49:56.732 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 10 librados: watch_flush enter 2024-08-09T14:49:56.732 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 10 librados: watch_flush exit 2024-08-09T14:49:56.732 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 20 client.4180.objecter shutdown clearing up homeless session... 2024-08-09T14:49:56.732 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 10 client.4180.objecter successfully canceled tick 2024-08-09T14:49:56.732 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 1 -- 172.21.15.155:0/1625076925 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fda4806d730 msgr2=0x7fda4806fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:56.732 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 1 --2- 172.21.15.155:0/1625076925 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fda4806d730 0x7fda4806fbf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fda60007fd0 tx=0x7fda60007e80 comp rx=0 tx=0).stop 2024-08-09T14:49:56.733 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 10 monclient: shutdown 2024-08-09T14:49:56.733 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:49:56.733 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 1 -- 172.21.15.155:0/1625076925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda640b4550 msgr2=0x7fda64143eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:49:56.733 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 1 --2- 172.21.15.155:0/1625076925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda640b4550 0x7fda64143eb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fda54034040 tx=0x7fda540046c0 comp rx=0 tx=0).stop 2024-08-09T14:49:56.733 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 1 -- 172.21.15.155:0/1625076925 shutdown_connections 2024-08-09T14:49:56.733 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 1 --2- 172.21.15.155:0/1625076925 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fda4806d730 0x7fda4806fbf0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:56.733 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 1 --2- 172.21.15.155:0/1625076925 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fda64142ce0 0x7fda641443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:56.733 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 1 --2- 172.21.15.155:0/1625076925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fda640b4550 0x7fda64143eb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:56.733 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 1 --2- 172.21.15.155:0/1625076925 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fda6406d740 0x7fda64146650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:49:56.733 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 1 -- 172.21.15.155:0/1625076925 >> 172.21.15.155:0/1625076925 conn(0x7fda640b8d30 msgr2=0x7fda640b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:49:56.733 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 1 -- 172.21.15.155:0/1625076925 shutdown_connections 2024-08-09T14:49:56.733 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 1 -- 172.21.15.155:0/1625076925 wait complete. 2024-08-09T14:49:56.734 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:49:56.730+0000 7fda6c31a640 1 librados: shutdown 2024-08-09T14:49:56.741 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-08-09T14:49:56.745 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:09.518 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.773 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:09.548 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.826 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:09.609 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.837 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:09.600 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.941 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:09.723 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:56.983 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:09.748 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.047 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:09.800 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.047 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:09.820 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.075 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:09.850 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.099 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:09.882 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.112 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:09.875 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.128 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:09.901 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.205 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:09.987 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.256 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:10.021 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.311 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:10.093 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.353 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:10.140 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.374 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:10.149 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.377 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:10.160 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.393 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:10.156 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.484 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:10.257 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.487 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:10.269 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.525 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:10.289 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.599 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:10.381 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.627 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:10.415 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.641 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:10.424 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.669 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:10.431 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.674 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:10.450 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.765 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:10.547 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.774 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:10.547 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.803 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:10.568 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.898 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:10.679 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.904 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:10.692 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.906 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:10.689 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.936 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:10.698 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:57.964 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:10.739 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.033 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:10.815 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.062 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:10.835 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.067 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:10.832 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.169 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:10.956 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.182 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:10.965 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.189 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:10.971 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.205 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:10.968 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.257 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:11.032 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.301 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:11.082 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.336 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:11.100 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.358 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:11.131 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.438 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:11.225 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.477 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:11.259 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.550 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:11.325 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.653 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:11.426 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.653 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:11.435 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.700 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:11.463 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.706 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:11.494 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.707 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:11.490 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.755 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:11.517 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.779 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:11.560 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.828 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:11.593 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.833 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:11.615 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.882 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:11.647 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.946 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:11.719 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:58.972 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:11.760 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:59.009 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:11.791 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:59.075 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:11.856 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:59.111 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:11.887 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:59.192 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:11.968 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:59.245 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:12.018 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:59.459 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:12.246 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:59.514 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:12.301 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:59.586 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:12.358 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:49:59.647 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:12.420 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:03.548 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-09T14:50:03.640 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.638+0000 7fd2fad0f640 10 monclient: get_monmap_and_config 2024-08-09T14:50:03.640 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.638+0000 7fd2fad0f640 10 monclient: build_initial_monmap 2024-08-09T14:50:03.640 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.638+0000 7fd2fad0f640 10 monclient: monmap: 2024-08-09T14:50:03.640 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:50:03.640 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:03.640 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:50:03.639983+0000 2024-08-09T14:50:03.640 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:50:03.639983+0000 2024-08-09T14:50:03.641 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:03.641 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:03.641 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:03.641 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:03.641 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:03.641 INFO:teuthology.misc.health.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:03.641 INFO:teuthology.misc.health.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:03.641 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:03.646 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.639+0000 7fd2fad0f640 1 Processor -- start 2024-08-09T14:50:03.646 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.639+0000 7fd2fad0f640 1 -- start start 2024-08-09T14:50:03.646 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.639+0000 7fd2fad0f640 10 monclient: init 2024-08-09T14:50:03.646 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.640+0000 7fd2fad0f640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:03.647 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.640+0000 7fd2fad0f640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-08-09T14:50:03.647 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.640+0000 7fd2fad0f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2f40b0190 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:03.647 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.640+0000 7fd2fad0f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2f40ae390 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:03.647 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.640+0000 7fd2fad0f640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd2f414a1e0 0x7fd2f414c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:03.647 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.640+0000 7fd2fad0f640 10 monclient(hunting): picked mon.noname-a con 0x7fd2f414a1e0 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:03.647 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.640+0000 7fd2fad0f640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2f40adea0 con 0x7fd2f40b0190 2024-08-09T14:50:03.647 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.640+0000 7fd2fad0f640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2f406ef30 con 0x7fd2f40ae390 2024-08-09T14:50:03.647 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.640+0000 7fd2fad0f640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:03.647 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.640+0000 7fd2fad0f640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7fd2f40b0570 con 0x7fd2f414a1e0 2024-08-09T14:50:03.647 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.640+0000 7fd2fad0f640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:03.647 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.640+0000 7fd2fad0f640 10 monclient(hunting): authenticate will time out at 1210.677510s 2024-08-09T14:50:03.647 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.641+0000 7fd2f3fff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fd2f40ae390 0x7fd2f413d980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.155:40872/0 (socket says 172.21.15.155:40872) 2024-08-09T14:50:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.641+0000 7fd2f92ac640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd2f414a1e0 0x7fd2f414c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.641+0000 7fd2f3fff640 1 -- 172.21.15.155:0/608142688 learned_addr learned my addr 172.21.15.155:0/608142688 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.642+0000 7fd2f92ac640 10 monclient(hunting): get_auth_request con 0x7fd2f414a1e0 auth_method 0 2024-08-09T14:50:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.642+0000 7fd2f92ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.642+0000 7fd2f92ac640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.642+0000 7fd2f92ac640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.642+0000 7fd2f92ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.642+0000 7fd2f92ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.642+0000 7fd2f92ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.645+0000 7fd2f92ac640 10 monclient(hunting): handle_auth_done global_id 4200 payload 274 2024-08-09T14:50:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.645+0000 7fd2f92ac640 1 -- 172.21.15.155:0/608142688 >> v1:172.21.15.194:6789/0 conn(0x7fd2f40ae390 legacy=0x7fd2f413d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.645+0000 7fd2f92ac640 1 -- 172.21.15.155:0/608142688 >> v1:172.21.15.155:6789/0 conn(0x7fd2f40b0190 legacy=0x7fd2f4143e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:03.648 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.645+0000 7fd2f92ac640 10 monclient: _finish_hunting 0 2024-08-09T14:50:03.649 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.645+0000 7fd2f92ac640 1 monclient: found mon.noname-a 2024-08-09T14:50:03.649 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.645+0000 7fd2f92ac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:03.649 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.645+0000 7fd2f92ac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:03.649 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.645+0000 7fd2f92ac640 1 -- 172.21.15.155:0/608142688 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f413e410 con 0x7fd2f414a1e0 2024-08-09T14:50:03.651 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.650+0000 7fd2f92ac640 1 --2- 172.21.15.155:0/608142688 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2f414a1e0 0x7fd2f414c5d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009d90 tx=0x7fd2ec02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9eeafae02e831eb server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:03.655 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2f37fe640 1 -- 172.21.15.155:0/608142688 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec039070 con 0x7fd2f414a1e0 2024-08-09T14:50:03.655 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:03.655 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:03.655 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2f37fe640 10 monclient: dump: 2024-08-09T14:50:03.655 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:03.656 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:03.656 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:03.656 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:03.656 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:03.656 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:03.656 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:03.656 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:03.656 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:03.656 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:03.656 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2f37fe640 10 monclient: _finish_auth 0 2024-08-09T14:50:03.656 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2f37fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:03.657 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:03.657 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2f37fe640 1 -- 172.21.15.155:0/608142688 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2ec02cae0 con 0x7fd2f414a1e0 2024-08-09T14:50:03.657 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2f37fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:03.657 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2f37fe640 1 -- 172.21.15.155:0/608142688 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2ec034440 con 0x7fd2f414a1e0 2024-08-09T14:50:03.657 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:03.657 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:03.657 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2f37fe640 10 monclient: dump: 2024-08-09T14:50:03.657 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:03.657 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:03.657 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:03.657 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:03.657 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:03.657 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:03.658 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:03.658 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:03.658 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:03.658 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:03.658 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2fad0f640 5 monclient: authenticate success, global_id 4200 2024-08-09T14:50:03.658 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2fad0f640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:03.658 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.654+0000 7fd2fad0f640 4 set_mon_vals no callback set 2024-08-09T14:50:03.665 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.655+0000 7fd2fad0f640 10 monclient: shutdown 2024-08-09T14:50:03.665 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.655+0000 7fd2fad0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:03.665 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.655+0000 7fd2fad0f640 1 -- 172.21.15.155:0/608142688 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2f414a1e0 msgr2=0x7fd2f414c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:03.665 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.655+0000 7fd2fad0f640 1 --2- 172.21.15.155:0/608142688 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2f414a1e0 0x7fd2f414c5d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec009d90 tx=0x7fd2ec02bdb0 comp rx=0 tx=0).stop 2024-08-09T14:50:03.665 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.655+0000 7fd2fad0f640 1 -- 172.21.15.155:0/608142688 shutdown_connections 2024-08-09T14:50:03.666 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.655+0000 7fd2fad0f640 1 --2- 172.21.15.155:0/608142688 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2f414a1e0 0x7fd2f414c5d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:03.666 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.655+0000 7fd2fad0f640 1 -- 172.21.15.155:0/608142688 >> 172.21.15.155:0/608142688 conn(0x7fd2f406bf50 msgr2=0x7fd2f406c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:03.666 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.655+0000 7fd2fad0f640 1 -- 172.21.15.155:0/608142688 shutdown_connections 2024-08-09T14:50:03.666 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.655+0000 7fd2fad0f640 1 -- 172.21.15.155:0/608142688 wait complete. 2024-08-09T14:50:03.666 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.656+0000 7fd2fad0f640 10 monclient: build_initial_monmap 2024-08-09T14:50:03.666 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.656+0000 7fd2fad0f640 10 monclient: monmap: 2024-08-09T14:50:03.666 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:50:03.666 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:03.666 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:03.666 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-08-09T14:50:03.666 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:03.667 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:03.667 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:03.667 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:03.667 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:03.667 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:03.667 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.657+0000 7fd2fad0f640 1 librados: starting msgr at 2024-08-09T14:50:03.667 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.657+0000 7fd2fad0f640 1 librados: starting objecter 2024-08-09T14:50:03.667 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.657+0000 7fd2fad0f640 1 Processor -- start 2024-08-09T14:50:03.667 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.657+0000 7fd2fad0f640 1 -- start start 2024-08-09T14:50:03.667 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.657+0000 7fd2fad0f640 1 librados: setting wanted keys 2024-08-09T14:50:03.667 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.657+0000 7fd2fad0f640 1 librados: calling monclient init 2024-08-09T14:50:03.667 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.657+0000 7fd2fad0f640 10 monclient: init 2024-08-09T14:50:03.667 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.658+0000 7fd2fad0f640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:03.667 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.658+0000 7fd2fad0f640 10 monclient: _add_conns ranks=[1,2,0] 2024-08-09T14:50:03.668 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.658+0000 7fd2fad0f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd2f40ae390 0x7fd2f4143660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:03.668 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.658+0000 7fd2fad0f640 10 monclient(hunting): picked mon.noname-b con 0x7fd2f40ae390 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:03.668 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.658+0000 7fd2fad0f640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd2f40b0190 0x7fd2f4140c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:03.668 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.658+0000 7fd2fad0f640 10 monclient(hunting): picked mon.noname-c con 0x7fd2f40b0190 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:03.668 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.658+0000 7fd2fad0f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2f414a1e0 0x7fd2f41346c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:03.668 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.658+0000 7fd2fad0f640 10 monclient(hunting): picked mon.noname-a con 0x7fd2f414a1e0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:03.668 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2fad0f640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:03.668 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2fad0f640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f4073e60 con 0x7fd2f414a1e0 2024-08-09T14:50:03.668 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2fad0f640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:03.668 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2fad0f640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd2f4073ce0 con 0x7fd2f40ae390 2024-08-09T14:50:03.668 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2fad0f640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:03.668 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2fad0f640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fd2f413e090 con 0x7fd2f40b0190 2024-08-09T14:50:03.668 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2fad0f640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:03.669 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2fad0f640 10 monclient(hunting): authenticate will time out at 1210.696127s 2024-08-09T14:50:03.669 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2f8aab640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd2f40ae390 0x7fd2f4143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:03.669 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2f3fff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd2f40b0190 0x7fd2f4140c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:03.669 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2f92ac640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2f414a1e0 0x7fd2f41346c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:03.669 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2f8aab640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd2f40ae390 0x7fd2f4143660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:60782/0 (socket says 172.21.15.155:60782) 2024-08-09T14:50:03.669 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2f8aab640 1 -- 172.21.15.155:0/4141995504 learned_addr learned my addr 172.21.15.155:0/4141995504 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:03.669 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2f8aab640 10 monclient(hunting): get_auth_request con 0x7fd2f40ae390 auth_method 0 2024-08-09T14:50:03.669 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2f8aab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:03.669 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2f8aab640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:03.669 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2f8aab640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:03.670 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2f3fff640 10 monclient(hunting): get_auth_request con 0x7fd2f40b0190 auth_method 0 2024-08-09T14:50:03.670 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:03.670 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2f3fff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:03.670 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.659+0000 7fd2f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:03.670 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.660+0000 7fd2f92ac640 10 monclient(hunting): get_auth_request con 0x7fd2f414a1e0 auth_method 0 2024-08-09T14:50:03.670 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.660+0000 7fd2f92ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:03.670 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.660+0000 7fd2f92ac640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:03.670 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.660+0000 7fd2f92ac640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:03.670 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.660+0000 7fd2f8aab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:03.671 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.660+0000 7fd2f8aab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:03.671 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.660+0000 7fd2f8aab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:03.671 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.660+0000 7fd2f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:03.671 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.660+0000 7fd2f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:03.671 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.660+0000 7fd2f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:03.671 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.660+0000 7fd2f92ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:03.671 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.660+0000 7fd2f92ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:03.671 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.660+0000 7fd2f92ac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:03.671 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.663+0000 7fd2f3fff640 10 monclient(hunting): handle_auth_done global_id 4190 payload 824 2024-08-09T14:50:03.671 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.663+0000 7fd2f3fff640 1 -- 172.21.15.155:0/4141995504 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd2f40ae390 msgr2=0x7fd2f4143660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:03.671 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.663+0000 7fd2f3fff640 1 --2- 172.21.15.155:0/4141995504 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd2f40ae390 0x7fd2f4143660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:03.671 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.663+0000 7fd2f3fff640 1 -- 172.21.15.155:0/4141995504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2f414a1e0 msgr2=0x7fd2f41346c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:03.671 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.663+0000 7fd2f3fff640 1 --2- 172.21.15.155:0/4141995504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2f414a1e0 0x7fd2f41346c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:03.672 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.663+0000 7fd2f3fff640 10 monclient: _finish_hunting 0 2024-08-09T14:50:03.672 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.663+0000 7fd2f3fff640 1 monclient: found mon.noname-c 2024-08-09T14:50:03.672 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.663+0000 7fd2f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:03.672 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.663+0000 7fd2f3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:50:03.672 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.663+0000 7fd2f3fff640 1 -- 172.21.15.155:0/4141995504 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2f41402d0 con 0x7fd2f40b0190 2024-08-09T14:50:03.673 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.665+0000 7fd2f3fff640 1 --2- 172.21.15.155:0/4141995504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd2f40b0190 0x7fd2f4140c40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000c780 tx=0x7fd2e000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:03.673 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.665+0000 7fd2f17fa640 10 client.?.objecter ms_handle_connect 0x7fd2f40b0190 2024-08-09T14:50:03.673 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.665+0000 7fd2f17fa640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:03.673 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2f17fa640 1 -- 172.21.15.155:0/4141995504 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0018070 con 0x7fd2f40b0190 2024-08-09T14:50:03.673 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:03.673 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2f17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:03.674 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2f17fa640 10 monclient: dump: 2024-08-09T14:50:03.674 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:03.674 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:03.674 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:03.674 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:03.674 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:03.674 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:03.674 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:03.674 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:03.674 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:03.674 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:03.674 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2f17fa640 10 monclient: _finish_auth 0 2024-08-09T14:50:03.675 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2f17fa640 10 monclient: _check_auth_tickets 2024-08-09T14:50:03.675 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:03.675 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2f17fa640 1 -- 172.21.15.155:0/4141995504 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2e00072d0 con 0x7fd2f40b0190 2024-08-09T14:50:03.675 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2f17fa640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:03.675 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2f17fa640 1 -- 172.21.15.155:0/4141995504 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2e00047e0 con 0x7fd2f40b0190 2024-08-09T14:50:03.675 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:03.675 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:03.675 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2f17fa640 10 monclient: dump: 2024-08-09T14:50:03.675 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:03.675 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:03.675 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:03.675 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:03.676 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:03.676 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:03.676 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:03.676 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:03.676 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:03.676 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2f2ffd640 4 set_mon_vals no callback set 2024-08-09T14:50:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2fad0f640 5 monclient: authenticate success, global_id 4190 2024-08-09T14:50:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2fad0f640 10 monclient: _renew_subs 2024-08-09T14:50:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2fad0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2fad0f640 1 -- 172.21.15.155:0/4141995504 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2f4140540 con 0x7fd2f40b0190 2024-08-09T14:50:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2fad0f640 10 client.4190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2fad0f640 10 monclient: _renew_subs 2024-08-09T14:50:03.676 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2fad0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:03.677 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2fad0f640 1 -- 172.21.15.155:0/4141995504 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2f4134c00 con 0x7fd2f40b0190 2024-08-09T14:50:03.677 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.670+0000 7fd2fad0f640 1 librados: init done 2024-08-09T14:50:03.677 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.671+0000 7fd2fad0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:03.677 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.671+0000 7fd2fad0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:03.677 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.671+0000 7fd2fad0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:03.677 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.671+0000 7fd2fad0f640 1 -- 172.21.15.155:0/4141995504 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2f4141180 con 0x7fd2f40b0190 2024-08-09T14:50:03.724 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.722+0000 7fd2f17fa640 1 -- 172.21.15.155:0/4141995504 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7fd2e0004980 con 0x7fd2f40b0190 2024-08-09T14:50:03.724 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.723+0000 7fd2f17fa640 1 --2- 172.21.15.155:0/4141995504 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fd2d806d730 0x7fd2d806fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:03.724 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.723+0000 7fd2f8aab640 1 --2- 172.21.15.155:0/4141995504 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fd2d806d730 0x7fd2d806fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:03.724 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.723+0000 7fd2f8aab640 10 monclient: get_auth_request con 0x7fd2d806d730 auth_method 0 2024-08-09T14:50:03.725 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.723+0000 7fd2f8aab640 1 --2- 172.21.15.155:0/4141995504 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fd2d806d730 0x7fd2d806fbf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8001840 tx=0x7fd2e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:03.725 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:03.723+0000 7fd2f17fa640 10 client.4190.objecter ms_handle_connect 0x7fd2d806d730 2024-08-09T14:50:04.037 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.035+0000 7fd2f17fa640 1 -- 172.21.15.155:0/4141995504 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fd2e0057a30 con 0x7fd2f40b0190 2024-08-09T14:50:04.037 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.036+0000 7fd2f17fa640 10 client.4190.objecter ms_dispatch 0x7fd2f4000b90 osd_map(10..10 src has 1..10) 2024-08-09T14:50:04.037 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.036+0000 7fd2f17fa640 3 client.4190.objecter handle_osd_map got epochs [10,10] > 0 2024-08-09T14:50:04.037 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.036+0000 7fd2f17fa640 3 client.4190.objecter handle_osd_map decoding full epoch 10 2024-08-09T14:50:04.038 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.036+0000 7fd2f17fa640 20 client.4190.objecter dump_active .. 0 homeless 2024-08-09T14:50:04.038 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.036+0000 7fd2f17fa640 1 -- 172.21.15.155:0/4141995504 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7fd2f4141180 con 0x7fd2f40b0190 2024-08-09T14:50:04.038 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.036+0000 7fd2f17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:04.038 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.036+0000 7fd2f17fa640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:04.259 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.258+0000 7fd2fad0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:04.259 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.258+0000 7fd2fad0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:04.259 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.258+0000 7fd2fad0f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:04.259 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.258+0000 7fd2fad0f640 1 -- 172.21.15.155:0/4141995504 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2f40b0c60 con 0x7fd2f40b0190 2024-08-09T14:50:04.288 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.286+0000 7fd2f17fa640 1 -- 172.21.15.155:0/4141995504 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fd2e0021070 con 0x7fd2f40b0190 2024-08-09T14:50:04.288 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.286+0000 7fd2f17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:04.288 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.286+0000 7fd2f17fa640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:04.288 INFO:teuthology.misc.health.smithi155.stdout: 2024-08-09T14:50:04.288 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"f3d21f4d-627f-4083-bf15-ae5f3fe632de","created":"2024-08-09T14:49:41.321598+0000","modified":"2024-08-09T14:49:46.978521+0000","last_up_change":"0.000000","last_in_change":"2024-08-09T14:49:46.978521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6526d474-661b-48e6-bcc7-6288c7477f78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"87673c5a-b124-419e-98da-17a7b7fbdf25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a14a708a-903f-4de2-8228-79e79ed135e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8e139d50-e25c-4428-9edf-c696189cb40e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0331386-2f50-4bf9-b72e-24c80f9fe9a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0f901e55-2ba8-4144-9802-3a109884254c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4b8215-814e-4b5a-9877-0ab63818b0ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"26ed84b6-639d-4ccd-bb1d-c64b593ada7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-09T14:50:04.290 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 10 librados: watch_flush enter 2024-08-09T14:50:04.290 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 10 librados: watch_flush exit 2024-08-09T14:50:04.290 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 20 client.4190.objecter shutdown clearing up homeless session... 2024-08-09T14:50:04.290 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 10 client.4190.objecter successfully canceled tick 2024-08-09T14:50:04.291 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 1 -- 172.21.15.155:0/4141995504 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fd2d806d730 msgr2=0x7fd2d806fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:04.291 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 1 --2- 172.21.15.155:0/4141995504 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fd2d806d730 0x7fd2d806fbf0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8001840 tx=0x7fd2e8009040 comp rx=0 tx=0).stop 2024-08-09T14:50:04.291 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 10 monclient: shutdown 2024-08-09T14:50:04.291 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:04.291 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 1 -- 172.21.15.155:0/4141995504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd2f40b0190 msgr2=0x7fd2f4140c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:04.291 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 1 --2- 172.21.15.155:0/4141995504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd2f40b0190 0x7fd2f4140c40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd2e000c780 tx=0x7fd2e000cc40 comp rx=0 tx=0).stop 2024-08-09T14:50:04.292 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 1 -- 172.21.15.155:0/4141995504 shutdown_connections 2024-08-09T14:50:04.292 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 1 --2- 172.21.15.155:0/4141995504 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fd2d806d730 0x7fd2d806fbf0 secure :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8001840 tx=0x7fd2e8009040 comp rx=0 tx=0).stop 2024-08-09T14:50:04.292 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 1 --2- 172.21.15.155:0/4141995504 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd2f414a1e0 0x7fd2f41346c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:04.292 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 1 --2- 172.21.15.155:0/4141995504 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd2f40b0190 0x7fd2f4140c40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:04.292 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 1 --2- 172.21.15.155:0/4141995504 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd2f40ae390 0x7fd2f4143660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:04.292 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 1 -- 172.21.15.155:0/4141995504 >> 172.21.15.155:0/4141995504 conn(0x7fd2f406bf50 msgr2=0x7fd2f40af660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:04.292 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 1 -- 172.21.15.155:0/4141995504 shutdown_connections 2024-08-09T14:50:04.292 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.289+0000 7fd2fad0f640 1 -- 172.21.15.155:0/4141995504 wait complete. 2024-08-09T14:50:04.292 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:04.290+0000 7fd2fad0f640 1 librados: shutdown 2024-08-09T14:50:04.304 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-08-09T14:50:04.523 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:17.305 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:04.577 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:17.341 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:04.843 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:17.606 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:05.095 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:17.877 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:05.128 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:17.909 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:05.181 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:17.956 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:05.311 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:18.099 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:05.649 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:18.422 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:10.168 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:22.933 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:10.173 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:22.939 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:10.337 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:23.119 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:10.343 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:23.126 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:10.555 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:23.318 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:10.559 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:23.322 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:10.713 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:23.495 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:10.719 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:23.501 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:10.844 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:23.619 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:10.848 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:23.623 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:10.855 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:23.637 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:10.860 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:23.641 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:11.110 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-09T14:50:11.198 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:23.985 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:11.202 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:23.989 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:11.203 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.202+0000 7f43ed819640 10 monclient: get_monmap_and_config 2024-08-09T14:50:11.204 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.202+0000 7f43ed819640 10 monclient: build_initial_monmap 2024-08-09T14:50:11.204 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.202+0000 7f43ed819640 10 monclient: monmap: 2024-08-09T14:50:11.204 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:50:11.204 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:11.204 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:50:11.203175+0000 2024-08-09T14:50:11.204 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:50:11.203175+0000 2024-08-09T14:50:11.204 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:11.204 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:11.204 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:11.204 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:11.205 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:11.205 INFO:teuthology.misc.health.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:11.205 INFO:teuthology.misc.health.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:11.205 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:11.205 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.202+0000 7f43ed819640 1 Processor -- start 2024-08-09T14:50:11.205 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.202+0000 7f43ed819640 1 -- start start 2024-08-09T14:50:11.205 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.202+0000 7f43ed819640 10 monclient: init 2024-08-09T14:50:11.207 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.203+0000 7f43ed819640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:11.207 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.203+0000 7f43ed819640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-08-09T14:50:11.208 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.203+0000 7f43ed819640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f43e806d740 0x7f43e813c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:11.208 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.203+0000 7f43ed819640 10 monclient(hunting): picked mon.noname-c con 0x7f43e806d740 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:11.208 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.203+0000 7f43ed819640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43e813cc20 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:11.208 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.203+0000 7f43ed819640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43e80b4c60 0x7f43e8142ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:11.208 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.203+0000 7f43ed819640 10 monclient(hunting): picked mon.noname-b con 0x7f43e80b4c60 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:11.208 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.203+0000 7f43ed819640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43e8143220 con 0x7f43e813cc20 2024-08-09T14:50:11.208 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.203+0000 7f43ed819640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:11.208 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.203+0000 7f43ed819640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f43e806db20 con 0x7f43e806d740 2024-08-09T14:50:11.208 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.203+0000 7f43ed819640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:11.208 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.203+0000 7f43ed819640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f43e80b5180 con 0x7f43e80b4c60 2024-08-09T14:50:11.208 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.203+0000 7f43ed819640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:11.208 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.203+0000 7f43ed819640 10 monclient(hunting): authenticate will time out at 1218.240657s 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.204+0000 7f43e77fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43e80b4c60 0x7f43e8142ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.204+0000 7f43e6ffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f43e806d740 0x7f43e813c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.204+0000 7f43e67fc640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f43e813cc20 0x7f43e80b4550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.155:55552/0 (socket says 172.21.15.155:55552) 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.204+0000 7f43e67fc640 1 -- 172.21.15.155:0/72344654 learned_addr learned my addr 172.21.15.155:0/72344654 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.204+0000 7f43e77fe640 10 monclient(hunting): get_auth_request con 0x7f43e80b4c60 auth_method 0 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.204+0000 7f43e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.204+0000 7f43e77fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.204+0000 7f43e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.205+0000 7f43e6ffd640 10 monclient(hunting): get_auth_request con 0x7f43e806d740 auth_method 0 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.205+0000 7f43e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.205+0000 7f43e6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.205+0000 7f43e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.205+0000 7f43e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.205+0000 7f43e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:11.209 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.205+0000 7f43e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:11.210 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.205+0000 7f43e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:11.210 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.205+0000 7f43e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:11.210 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.205+0000 7f43e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:11.211 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.207+0000 7f43e77fe640 10 monclient(hunting): handle_auth_done global_id 4193 payload 274 2024-08-09T14:50:11.211 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.207+0000 7f43e77fe640 1 -- 172.21.15.155:0/72344654 >> v2:172.21.15.194:3300/0 conn(0x7f43e806d740 msgr2=0x7f43e813c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:11.211 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.207+0000 7f43e77fe640 1 --2- 172.21.15.155:0/72344654 >> v2:172.21.15.194:3300/0 conn(0x7f43e806d740 0x7f43e813c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:11.211 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.207+0000 7f43e77fe640 1 -- 172.21.15.155:0/72344654 >> v1:172.21.15.194:6789/0 conn(0x7f43e813cc20 legacy=0x7f43e80b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:11.211 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.207+0000 7f43e77fe640 10 monclient: _finish_hunting 0 2024-08-09T14:50:11.211 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.207+0000 7f43e77fe640 1 monclient: found mon.noname-b 2024-08-09T14:50:11.211 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.207+0000 7f43e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:11.211 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.207+0000 7f43e77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:11.211 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.207+0000 7f43e77fe640 1 -- 172.21.15.155:0/72344654 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e81433f0 con 0x7f43e80b4c60 2024-08-09T14:50:11.211 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.209+0000 7f43e77fe640 1 --2- 172.21.15.155:0/72344654 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43e80b4c60 0x7f43e8142ce0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f43e0009d90 tx=0x7f43e002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3d88261a751d3e server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:11.215 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.213+0000 7f43e5ffb640 1 -- 172.21.15.155:0/72344654 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e0039070 con 0x7f43e80b4c60 2024-08-09T14:50:11.215 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:11.215 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43e5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:11.215 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43e5ffb640 10 monclient: dump: 2024-08-09T14:50:11.215 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:11.215 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:11.215 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:11.215 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:11.215 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:11.215 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:11.215 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43e5ffb640 10 monclient: _finish_auth 0 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43e5ffb640 10 monclient: _check_auth_tickets 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43e5ffb640 1 -- 172.21.15.155:0/72344654 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e002cbc0 con 0x7f43e80b4c60 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43e5ffb640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43e5ffb640 1 -- 172.21.15.155:0/72344654 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e0034440 con 0x7f43e80b4c60 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43e5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43e5ffb640 10 monclient: dump: 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:11.216 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:11.217 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:11.217 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:11.217 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:11.217 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:11.217 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:11.217 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:11.217 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:11.217 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43ed819640 5 monclient: authenticate success, global_id 4193 2024-08-09T14:50:11.217 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43ed819640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:11.217 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43ed819640 4 set_mon_vals no callback set 2024-08-09T14:50:11.221 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43ed819640 10 monclient: shutdown 2024-08-09T14:50:11.221 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43ed819640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:11.221 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43ed819640 1 -- 172.21.15.155:0/72344654 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43e80b4c60 msgr2=0x7f43e8142ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:11.221 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43ed819640 1 --2- 172.21.15.155:0/72344654 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43e80b4c60 0x7f43e8142ce0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f43e0009d90 tx=0x7f43e002be10 comp rx=0 tx=0).stop 2024-08-09T14:50:11.221 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43ed819640 1 -- 172.21.15.155:0/72344654 shutdown_connections 2024-08-09T14:50:11.221 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43ed819640 1 --2- 172.21.15.155:0/72344654 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43e80b4c60 0x7f43e8142ce0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:11.221 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43ed819640 1 --2- 172.21.15.155:0/72344654 >> v2:172.21.15.194:3300/0 conn(0x7f43e806d740 0x7f43e813c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:11.221 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.214+0000 7f43ed819640 1 -- 172.21.15.155:0/72344654 >> 172.21.15.155:0/72344654 conn(0x7f43e80b8d30 msgr2=0x7f43e80b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:11.222 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.215+0000 7f43ed819640 1 -- 172.21.15.155:0/72344654 shutdown_connections 2024-08-09T14:50:11.222 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.215+0000 7f43ed819640 1 -- 172.21.15.155:0/72344654 wait complete. 2024-08-09T14:50:11.222 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.215+0000 7f43ed819640 10 monclient: build_initial_monmap 2024-08-09T14:50:11.222 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.215+0000 7f43ed819640 10 monclient: monmap: 2024-08-09T14:50:11.222 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:50:11.222 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:11.222 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:11.222 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-08-09T14:50:11.222 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:11.222 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:11.222 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:11.222 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:11.223 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:11.223 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:11.223 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.216+0000 7f43ed819640 1 librados: starting msgr at 2024-08-09T14:50:11.223 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.216+0000 7f43ed819640 1 librados: starting objecter 2024-08-09T14:50:11.223 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.216+0000 7f43ed819640 1 Processor -- start 2024-08-09T14:50:11.223 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.216+0000 7f43ed819640 1 -- start start 2024-08-09T14:50:11.223 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.216+0000 7f43ed819640 1 librados: setting wanted keys 2024-08-09T14:50:11.223 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.216+0000 7f43ed819640 1 librados: calling monclient init 2024-08-09T14:50:11.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.216+0000 7f43ed819640 10 monclient: init 2024-08-09T14:50:11.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:11.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 10 monclient: _add_conns ranks=[1,2,0] 2024-08-09T14:50:11.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43e806d740 0x7f43e813bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:11.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 10 monclient(hunting): picked mon.noname-b con 0x7f43e806d740 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:11.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43e813cc20 0x7f43e81dbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:11.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 10 monclient(hunting): picked mon.noname-c con 0x7f43e813cc20 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:11.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43e81dc200 0x7f43e81326d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:11.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 10 monclient(hunting): picked mon.noname-a con 0x7f43e81dc200 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:11.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:11.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f43e814d5c0 con 0x7f43e81dc200 2024-08-09T14:50:11.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:11.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f43e814d440 con 0x7f43e806d740 2024-08-09T14:50:11.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:11.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f43e814d740 con 0x7f43e813cc20 2024-08-09T14:50:11.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:11.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43ed819640 10 monclient(hunting): authenticate will time out at 1218.254208s 2024-08-09T14:50:11.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43e6ffd640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43e806d740 0x7f43e813bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:11.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43e6ffd640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43e806d740 0x7f43e813bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:34690/0 (socket says 172.21.15.155:34690) 2024-08-09T14:50:11.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43e6ffd640 1 -- 172.21.15.155:0/1657619042 learned_addr learned my addr 172.21.15.155:0/1657619042 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:11.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43e6ffd640 10 monclient(hunting): get_auth_request con 0x7f43e806d740 auth_method 0 2024-08-09T14:50:11.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:11.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43e6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:11.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:11.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.217+0000 7f43e77fe640 1 --2- 172.21.15.155:0/1657619042 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43e81dc200 0x7f43e81326d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:11.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e67fc640 1 --2- 172.21.15.155:0/1657619042 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43e813cc20 0x7f43e81dbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:11.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:11.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:11.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:11.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e77fe640 10 monclient(hunting): get_auth_request con 0x7f43e81dc200 auth_method 0 2024-08-09T14:50:11.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:11.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e77fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:11.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:11.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e67fc640 10 monclient(hunting): get_auth_request con 0x7f43e813cc20 auth_method 0 2024-08-09T14:50:11.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:11.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e67fc640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:11.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:11.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:11.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:11.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.218+0000 7f43e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:11.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.219+0000 7f43e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:11.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.219+0000 7f43e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:11.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.219+0000 7f43e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:11.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.221+0000 7f43e6ffd640 10 monclient(hunting): handle_auth_done global_id 4195 payload 824 2024-08-09T14:50:11.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.221+0000 7f43e6ffd640 1 -- 172.21.15.155:0/1657619042 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43e813cc20 msgr2=0x7f43e81dbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:11.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.221+0000 7f43e6ffd640 1 --2- 172.21.15.155:0/1657619042 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43e813cc20 0x7f43e81dbcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:11.228 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.221+0000 7f43e6ffd640 1 -- 172.21.15.155:0/1657619042 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43e81dc200 msgr2=0x7f43e81326d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:11.228 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.221+0000 7f43e6ffd640 1 --2- 172.21.15.155:0/1657619042 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43e81dc200 0x7f43e81326d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:11.228 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.221+0000 7f43e6ffd640 10 monclient: _finish_hunting 0 2024-08-09T14:50:11.228 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.221+0000 7f43e6ffd640 1 monclient: found mon.noname-b 2024-08-09T14:50:11.228 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.221+0000 7f43e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:11.228 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.221+0000 7f43e6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:11.228 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.221+0000 7f43e6ffd640 1 -- 172.21.15.155:0/1657619042 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43e813c470 con 0x7f43e806d740 2024-08-09T14:50:11.228 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.224+0000 7f43e6ffd640 1 --2- 172.21.15.155:0/1657619042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43e806d740 0x7f43e813bf30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f43d800e9e0 tx=0x7f43d800eea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:11.228 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.224+0000 7f43cffff640 10 client.?.objecter ms_handle_connect 0x7f43e806d740 2024-08-09T14:50:11.228 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.224+0000 7f43cffff640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:11.231 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43cffff640 1 -- 172.21.15.155:0/1657619042 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d8017070 con 0x7f43e806d740 2024-08-09T14:50:11.231 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:11.231 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43cffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:11.231 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43cffff640 10 monclient: dump: 2024-08-09T14:50:11.231 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:11.231 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:11.231 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:11.232 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:11.232 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:11.232 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:11.232 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:11.232 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:11.232 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:11.232 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:11.232 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43cffff640 10 monclient: _finish_auth 0 2024-08-09T14:50:11.232 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43cffff640 10 monclient: _check_auth_tickets 2024-08-09T14:50:11.232 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:11.232 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43cffff640 1 -- 172.21.15.155:0/1657619042 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43d800c9d0 con 0x7f43e806d740 2024-08-09T14:50:11.232 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43cffff640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:11.232 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43cffff640 1 -- 172.21.15.155:0/1657619042 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d800cdc0 con 0x7f43e806d740 2024-08-09T14:50:11.233 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43ed819640 5 monclient: authenticate success, global_id 4195 2024-08-09T14:50:11.233 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43ed819640 10 monclient: _renew_subs 2024-08-09T14:50:11.233 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43ed819640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:11.233 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43ed819640 1 -- 172.21.15.155:0/1657619042 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43e813e1a0 con 0x7f43e806d740 2024-08-09T14:50:11.233 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:11.233 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:11.233 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43cffff640 10 monclient: dump: 2024-08-09T14:50:11.233 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:11.233 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:11.233 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:11.234 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:11.234 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:11.234 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:11.234 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:11.234 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:11.234 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:11.234 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:11.234 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43e57fa640 4 set_mon_vals no callback set 2024-08-09T14:50:11.234 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43ed819640 10 client.4195.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:11.234 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43ed819640 10 monclient: _renew_subs 2024-08-09T14:50:11.234 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43ed819640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:11.234 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43ed819640 1 -- 172.21.15.155:0/1657619042 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43e813ffc0 con 0x7f43e806d740 2024-08-09T14:50:11.235 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.229+0000 7f43ed819640 1 librados: init done 2024-08-09T14:50:11.235 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.230+0000 7f43ed819640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:11.235 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.230+0000 7f43ed819640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:11.235 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.230+0000 7f43ed819640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:11.235 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.230+0000 7f43ed819640 1 -- 172.21.15.155:0/1657619042 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43e813c6e0 con 0x7f43e806d740 2024-08-09T14:50:11.261 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.260+0000 7f43cffff640 1 -- 172.21.15.155:0/1657619042 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f43d801a070 con 0x7f43e806d740 2024-08-09T14:50:11.261 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.260+0000 7f43cffff640 1 --2- 172.21.15.155:0/1657619042 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f43d406d730 0x7f43d406fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:11.262 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.261+0000 7f43e67fc640 1 --2- 172.21.15.155:0/1657619042 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f43d406d730 0x7f43d406fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:11.262 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.261+0000 7f43e67fc640 10 monclient: get_auth_request con 0x7f43d406d730 auth_method 0 2024-08-09T14:50:11.262 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.261+0000 7f43cffff640 10 client.4195.objecter ms_handle_connect 0x7f43d406d730 2024-08-09T14:50:11.262 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.261+0000 7f43e67fc640 1 --2- 172.21.15.155:0/1657619042 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f43d406d730 0x7f43d406fbf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f43e813ff70 tx=0x7f43dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:11.265 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.264+0000 7f43cffff640 1 -- 172.21.15.155:0/1657619042 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f43d80585f0 con 0x7f43e806d740 2024-08-09T14:50:11.265 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.264+0000 7f43cffff640 10 client.4195.objecter ms_dispatch 0x7f43e8000b90 osd_map(10..10 src has 1..10) 2024-08-09T14:50:11.265 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.264+0000 7f43cffff640 3 client.4195.objecter handle_osd_map got epochs [10,10] > 0 2024-08-09T14:50:11.265 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.264+0000 7f43cffff640 3 client.4195.objecter handle_osd_map decoding full epoch 10 2024-08-09T14:50:11.265 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.264+0000 7f43cffff640 20 client.4195.objecter dump_active .. 0 homeless 2024-08-09T14:50:11.461 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:24.234 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:11.467 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:24.240 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:11.658 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.656+0000 7f43cffff640 1 -- 172.21.15.155:0/1657619042 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f43d8059960 con 0x7f43e806d740 2024-08-09T14:50:11.658 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.657+0000 7f43cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:11.658 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.657+0000 7f43cffff640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:11.893 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.892+0000 7f43ed819640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:11.893 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.892+0000 7f43ed819640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:11.893 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.892+0000 7f43ed819640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:11.894 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.892+0000 7f43ed819640 1 -- 172.21.15.155:0/1657619042 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f43e806e840 con 0x7f43e806d740 2024-08-09T14:50:11.916 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.915+0000 7f43cffff640 1 -- 172.21.15.155:0/1657619042 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f43d805dc60 con 0x7f43e806d740 2024-08-09T14:50:11.916 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.915+0000 7f43cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:11.916 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.915+0000 7f43cffff640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:11.917 INFO:teuthology.misc.health.smithi155.stdout: 2024-08-09T14:50:11.917 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"f3d21f4d-627f-4083-bf15-ae5f3fe632de","created":"2024-08-09T14:49:41.321598+0000","modified":"2024-08-09T14:49:46.978521+0000","last_up_change":"0.000000","last_in_change":"2024-08-09T14:49:46.978521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6526d474-661b-48e6-bcc7-6288c7477f78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"87673c5a-b124-419e-98da-17a7b7fbdf25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a14a708a-903f-4de2-8228-79e79ed135e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8e139d50-e25c-4428-9edf-c696189cb40e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0331386-2f50-4bf9-b72e-24c80f9fe9a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0f901e55-2ba8-4144-9802-3a109884254c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4b8215-814e-4b5a-9877-0ab63818b0ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"26ed84b6-639d-4ccd-bb1d-c64b593ada7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-09T14:50:11.919 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 10 librados: watch_flush enter 2024-08-09T14:50:11.919 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 10 librados: watch_flush exit 2024-08-09T14:50:11.919 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 20 client.4195.objecter shutdown clearing up homeless session... 2024-08-09T14:50:11.919 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 10 client.4195.objecter successfully canceled tick 2024-08-09T14:50:11.919 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 1 -- 172.21.15.155:0/1657619042 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f43d406d730 msgr2=0x7f43d406fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:11.919 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 1 --2- 172.21.15.155:0/1657619042 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f43d406d730 0x7f43d406fbf0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f43e813ff70 tx=0x7f43dc009040 comp rx=0 tx=0).stop 2024-08-09T14:50:11.919 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 10 monclient: shutdown 2024-08-09T14:50:11.920 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:11.920 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 1 -- 172.21.15.155:0/1657619042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43e806d740 msgr2=0x7f43e813bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:11.920 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 1 --2- 172.21.15.155:0/1657619042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43e806d740 0x7f43e813bf30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f43d800e9e0 tx=0x7f43d800eea0 comp rx=0 tx=0).stop 2024-08-09T14:50:11.920 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 1 -- 172.21.15.155:0/1657619042 shutdown_connections 2024-08-09T14:50:11.920 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 1 --2- 172.21.15.155:0/1657619042 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f43d406d730 0x7f43d406fbf0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:11.920 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 1 --2- 172.21.15.155:0/1657619042 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f43e81dc200 0x7f43e81326d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:11.920 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 1 --2- 172.21.15.155:0/1657619042 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f43e813cc20 0x7f43e81dbcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:11.920 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 1 --2- 172.21.15.155:0/1657619042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43e806d740 0x7f43e813bf30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:11.920 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 1 -- 172.21.15.155:0/1657619042 >> 172.21.15.155:0/1657619042 conn(0x7f43e80b8d30 msgr2=0x7f43e80b9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:11.920 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 1 -- 172.21.15.155:0/1657619042 shutdown_connections 2024-08-09T14:50:11.920 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 1 -- 172.21.15.155:0/1657619042 wait complete. 2024-08-09T14:50:11.921 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:11.917+0000 7f43ed819640 1 librados: shutdown 2024-08-09T14:50:11.931 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-08-09T14:50:14.450 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:00:27.215 35989== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:14.861 INFO:tasks.ceph.osd.4.smithi194.stderr:==00:00:00:27.644 35683== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:14.980 INFO:tasks.ceph.osd.6.smithi194.stderr:==00:00:00:27.743 35686== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:15.228 INFO:tasks.ceph.osd.2.smithi155.stderr:==00:00:00:28.009 35987== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:15.404 INFO:tasks.ceph.osd.7.smithi194.stderr:==00:00:00:28.180 35685== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:15.510 INFO:tasks.ceph.osd.1.smithi155.stderr:==00:00:00:28.298 35986== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:15.648 INFO:tasks.ceph.osd.5.smithi194.stderr:==00:00:00:28.429 35684== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:15.978 INFO:tasks.ceph.osd.0.smithi155.stderr:==00:00:00:28.751 35988== Warning: unimplemented fcntl command: 1038 2024-08-09T14:50:18.738 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-09T14:50:18.841 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.839+0000 7f75fd330640 10 monclient: get_monmap_and_config 2024-08-09T14:50:18.842 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.839+0000 7f75fd330640 10 monclient: build_initial_monmap 2024-08-09T14:50:18.842 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.839+0000 7f75fd330640 10 monclient: monmap: 2024-08-09T14:50:18.842 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:50:18.842 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:18.842 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:50:18.840725+0000 2024-08-09T14:50:18.842 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:50:18.840725+0000 2024-08-09T14:50:18.842 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:18.842 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:18.842 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:18.842 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:18.843 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:18.843 INFO:teuthology.misc.health.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:18.843 INFO:teuthology.misc.health.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:18.843 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:18.843 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.840+0000 7f75fd330640 1 Processor -- start 2024-08-09T14:50:18.843 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.840+0000 7f75fd330640 1 -- start start 2024-08-09T14:50:18.843 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.840+0000 7f75fd330640 10 monclient: init 2024-08-09T14:50:18.845 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.841+0000 7f75fd330640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:18.845 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.841+0000 7f75fd330640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-08-09T14:50:18.845 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.841+0000 7f75fd330640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f75f806d740 0x7f75f813c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:18.845 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.841+0000 7f75fd330640 10 monclient(hunting): picked mon.noname-c con 0x7f75f806d740 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:18.845 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.841+0000 7f75fd330640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f813cc20 0x7f75f80b4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:18.845 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.841+0000 7f75fd330640 10 monclient(hunting): picked mon.noname-b con 0x7f75f813cc20 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.841+0000 7f75fd330640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75f80b4a90 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.841+0000 7f75fd330640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75f81433f0 con 0x7f75f80b4a90 2024-08-09T14:50:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.841+0000 7f75fd330640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.841+0000 7f75fd330640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f75f806db20 con 0x7f75f806d740 2024-08-09T14:50:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.841+0000 7f75fd330640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.841+0000 7f75fd330640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f75f80b5180 con 0x7f75f813cc20 2024-08-09T14:50:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.841+0000 7f75fd330640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.841+0000 7f75fd330640 10 monclient(hunting): authenticate will time out at 1225.878392s 2024-08-09T14:50:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.842+0000 7f75f67fc640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f813cc20 0x7f75f80b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.843+0000 7f75f67fc640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f813cc20 0x7f75f80b4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:59300/0 (socket says 172.21.15.155:59300) 2024-08-09T14:50:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.843+0000 7f75f67fc640 1 -- 172.21.15.155:0/1386877732 learned_addr learned my addr 172.21.15.155:0/1386877732 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:18.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.843+0000 7f75f67fc640 10 monclient(hunting): get_auth_request con 0x7f75f813cc20 auth_method 0 2024-08-09T14:50:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.843+0000 7f75f67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.843+0000 7f75f67fc640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.843+0000 7f75f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.843+0000 7f75f67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.843+0000 7f75f67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.843+0000 7f75f67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.846+0000 7f75f67fc640 10 monclient(hunting): handle_auth_done global_id 4199 payload 274 2024-08-09T14:50:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.846+0000 7f75f67fc640 1 -- 172.21.15.155:0/1386877732 >> v2:172.21.15.194:3300/0 conn(0x7f75f806d740 msgr2=0x7f75f813c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:18.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.846+0000 7f75f67fc640 1 --2- 172.21.15.155:0/1386877732 >> v2:172.21.15.194:3300/0 conn(0x7f75f806d740 0x7f75f813c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:18.848 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.846+0000 7f75f67fc640 1 -- 172.21.15.155:0/1386877732 >> v1:172.21.15.194:6789/0 conn(0x7f75f80b4a90 legacy=0x7f75f8142ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:18.848 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.846+0000 7f75f67fc640 10 monclient: _finish_hunting 0 2024-08-09T14:50:18.848 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.846+0000 7f75f67fc640 1 monclient: found mon.noname-b 2024-08-09T14:50:18.848 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.846+0000 7f75f67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:18.848 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.846+0000 7f75f67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:18.848 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.846+0000 7f75f67fc640 1 -- 172.21.15.155:0/1386877732 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f81435c0 con 0x7f75f813cc20 2024-08-09T14:50:18.851 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.850+0000 7f75f67fc640 1 --2- 172.21.15.155:0/1386877732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f813cc20 0x7f75f80b4550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009870 tx=0x7f75ec02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3da9bb23adfccb7 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:18.941 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75f5ffb640 1 -- 172.21.15.155:0/1386877732 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec0317c0 con 0x7f75f813cc20 2024-08-09T14:50:18.941 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:18.941 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75f5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:18.941 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75f5ffb640 10 monclient: dump: 2024-08-09T14:50:18.941 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:18.942 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:18.942 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:18.942 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:18.942 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:18.942 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:18.942 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:18.942 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:18.942 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:18.942 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:18.942 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75f5ffb640 10 monclient: _finish_auth 0 2024-08-09T14:50:18.942 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75f5ffb640 10 monclient: _check_auth_tickets 2024-08-09T14:50:18.942 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:18.943 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75f5ffb640 1 -- 172.21.15.155:0/1386877732 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75ec037070 con 0x7f75f813cc20 2024-08-09T14:50:18.943 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75fd330640 5 monclient: authenticate success, global_id 4199 2024-08-09T14:50:18.943 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75fd330640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:18.943 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75f5ffb640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:18.943 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75fd330640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:18.943 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75f5ffb640 1 -- 172.21.15.155:0/1386877732 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ec031b80 con 0x7f75f813cc20 2024-08-09T14:50:18.943 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75fd330640 4 set_mon_vals no callback set 2024-08-09T14:50:18.943 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:18.943 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:18.943 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75f5ffb640 10 monclient: dump: 2024-08-09T14:50:18.943 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:18.943 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:18.943 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:18.943 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:18.944 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:18.944 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:18.944 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:18.944 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:18.944 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:18.944 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:18.944 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75fd330640 10 monclient: shutdown 2024-08-09T14:50:18.944 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75fd330640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:18.944 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75fd330640 1 -- 172.21.15.155:0/1386877732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f813cc20 msgr2=0x7f75f80b4550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:18.944 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.939+0000 7f75fd330640 1 --2- 172.21.15.155:0/1386877732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f813cc20 0x7f75f80b4550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f75ec009870 tx=0x7f75ec02b860 comp rx=0 tx=0).stop 2024-08-09T14:50:18.944 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.940+0000 7f75fd330640 1 -- 172.21.15.155:0/1386877732 shutdown_connections 2024-08-09T14:50:18.944 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.940+0000 7f75fd330640 1 --2- 172.21.15.155:0/1386877732 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f813cc20 0x7f75f80b4550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:18.944 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.940+0000 7f75fd330640 1 --2- 172.21.15.155:0/1386877732 >> v2:172.21.15.194:3300/0 conn(0x7f75f806d740 0x7f75f813c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:18.945 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.940+0000 7f75fd330640 1 -- 172.21.15.155:0/1386877732 >> 172.21.15.155:0/1386877732 conn(0x7f75f80b8d30 msgr2=0x7f75f80b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:18.945 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.940+0000 7f75fd330640 1 -- 172.21.15.155:0/1386877732 shutdown_connections 2024-08-09T14:50:18.945 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.940+0000 7f75fd330640 1 -- 172.21.15.155:0/1386877732 wait complete. 2024-08-09T14:50:18.948 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.940+0000 7f75fd330640 10 monclient: build_initial_monmap 2024-08-09T14:50:18.948 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.940+0000 7f75fd330640 10 monclient: monmap: 2024-08-09T14:50:18.948 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:50:18.948 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:18.948 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:18.948 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-08-09T14:50:18.948 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:18.948 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:18.949 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:18.949 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:18.949 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:18.949 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:18.949 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.941+0000 7f75fd330640 1 librados: starting msgr at 2024-08-09T14:50:18.949 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.941+0000 7f75fd330640 1 librados: starting objecter 2024-08-09T14:50:18.949 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.941+0000 7f75fd330640 1 Processor -- start 2024-08-09T14:50:18.949 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.941+0000 7f75fd330640 1 -- start start 2024-08-09T14:50:18.949 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.941+0000 7f75fd330640 1 librados: setting wanted keys 2024-08-09T14:50:18.949 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.941+0000 7f75fd330640 1 librados: calling monclient init 2024-08-09T14:50:18.950 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.941+0000 7f75fd330640 10 monclient: init 2024-08-09T14:50:18.950 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:18.950 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 10 monclient: _add_conns ranks=[1,0,2] 2024-08-09T14:50:18.950 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75f806d740 0x7f75f8070140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:18.950 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 10 monclient(hunting): picked mon.noname-b con 0x7f75f806d740 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:18.950 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75f80b4a90 0x7f75f8135e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:18.950 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 10 monclient(hunting): picked mon.noname-a con 0x7f75f80b4a90 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:18.950 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f813cc20 0x7f75f81363d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:18.950 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 10 monclient(hunting): picked mon.noname-c con 0x7f75f813cc20 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:18.951 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:18.951 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f75f806edc0 con 0x7f75f80b4a90 2024-08-09T14:50:18.951 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:18.951 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f75f806ec40 con 0x7f75f806d740 2024-08-09T14:50:18.951 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:18.951 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f75f806ef40 con 0x7f75f813cc20 2024-08-09T14:50:18.951 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:18.951 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75fd330640 10 monclient(hunting): authenticate will time out at 1225.979619s 2024-08-09T14:50:18.951 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.942+0000 7f75f6ffd640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75f806d740 0x7f75f8070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:18.951 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.943+0000 7f75f6ffd640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75f806d740 0x7f75f8070140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:50028/0 (socket says 172.21.15.155:50028) 2024-08-09T14:50:18.951 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.943+0000 7f75f6ffd640 1 -- 172.21.15.155:0/427128302 learned_addr learned my addr 172.21.15.155:0/427128302 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:18.952 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.943+0000 7f75f6ffd640 10 monclient(hunting): get_auth_request con 0x7f75f806d740 auth_method 0 2024-08-09T14:50:18.952 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.943+0000 7f75f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:18.952 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.943+0000 7f75f6ffd640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:18.952 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.943+0000 7f75f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:18.952 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.943+0000 7f75f67fc640 1 --2- 172.21.15.155:0/427128302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75f80b4a90 0x7f75f8135e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:18.952 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.943+0000 7f75f67fc640 10 monclient(hunting): get_auth_request con 0x7f75f80b4a90 auth_method 0 2024-08-09T14:50:18.952 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.943+0000 7f75f67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:18.952 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.943+0000 7f75f67fc640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:18.952 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.943+0000 7f75f67fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:18.952 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.943+0000 7f75f67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:18.952 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.943+0000 7f75f67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:18.952 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.943+0000 7f75f77fe640 1 --2- 172.21.15.155:0/427128302 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f813cc20 0x7f75f81363d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:18.952 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.944+0000 7f75f67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:18.953 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.944+0000 7f75f6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:18.953 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.944+0000 7f75f6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:18.953 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.944+0000 7f75f6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:18.953 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.944+0000 7f75f77fe640 10 monclient(hunting): get_auth_request con 0x7f75f813cc20 auth_method 0 2024-08-09T14:50:18.953 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.944+0000 7f75f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:18.953 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.944+0000 7f75f77fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:18.953 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.944+0000 7f75f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:18.953 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.945+0000 7f75f77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:18.953 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.945+0000 7f75f77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:18.953 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.945+0000 7f75f77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:18.956 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.946+0000 7f75f6ffd640 10 monclient(hunting): handle_auth_done global_id 4198 payload 824 2024-08-09T14:50:18.956 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.946+0000 7f75f6ffd640 1 -- 172.21.15.155:0/427128302 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f813cc20 msgr2=0x7f75f81363d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:18.956 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.946+0000 7f75f6ffd640 1 --2- 172.21.15.155:0/427128302 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f813cc20 0x7f75f81363d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:18.956 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.946+0000 7f75f6ffd640 1 -- 172.21.15.155:0/427128302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75f80b4a90 msgr2=0x7f75f8135e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:18.956 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.946+0000 7f75f6ffd640 1 --2- 172.21.15.155:0/427128302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75f80b4a90 0x7f75f8135e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:18.956 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.946+0000 7f75f6ffd640 10 monclient: _finish_hunting 0 2024-08-09T14:50:18.956 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.946+0000 7f75f6ffd640 1 monclient: found mon.noname-b 2024-08-09T14:50:18.956 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.946+0000 7f75f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:18.957 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.946+0000 7f75f6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:18.957 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.946+0000 7f75f6ffd640 1 -- 172.21.15.155:0/427128302 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75f8071f70 con 0x7f75f806d740 2024-08-09T14:50:18.957 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.949+0000 7f75f6ffd640 1 --2- 172.21.15.155:0/427128302 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75f806d740 0x7f75f8070140 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f75e800b4a0 tx=0x7f75e800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:18.957 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.949+0000 7f75df7fe640 10 client.?.objecter ms_handle_connect 0x7f75f806d740 2024-08-09T14:50:18.957 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.949+0000 7f75df7fe640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:18.957 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75df7fe640 1 -- 172.21.15.155:0/427128302 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e8014070 con 0x7f75f806d740 2024-08-09T14:50:18.957 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:18.957 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:18.957 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75df7fe640 10 monclient: dump: 2024-08-09T14:50:18.957 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:18.958 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:18.958 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:18.958 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:18.958 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:18.958 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:18.958 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:18.958 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:18.958 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:18.958 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:18.958 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75df7fe640 10 monclient: _finish_auth 0 2024-08-09T14:50:18.958 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75df7fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:18.958 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:18.959 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75fd330640 5 monclient: authenticate success, global_id 4198 2024-08-09T14:50:18.959 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75fd330640 10 monclient: _renew_subs 2024-08-09T14:50:18.959 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75fd330640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:18.959 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75fd330640 1 -- 172.21.15.155:0/427128302 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75f813d1c0 con 0x7f75f806d740 2024-08-09T14:50:18.959 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75fd330640 10 client.4198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:18.959 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75fd330640 10 monclient: _renew_subs 2024-08-09T14:50:18.959 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75fd330640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:18.959 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75fd330640 1 -- 172.21.15.155:0/427128302 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75f8070cb0 con 0x7f75f806d740 2024-08-09T14:50:18.959 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.952+0000 7f75fd330640 1 librados: init done 2024-08-09T14:50:18.959 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.953+0000 7f75fd330640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:18.959 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.953+0000 7f75fd330640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:18.959 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.953+0000 7f75fd330640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:18.960 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:18.953+0000 7f75fd330640 1 -- 172.21.15.155:0/427128302 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75f8069630 con 0x7f75f806d740 2024-08-09T14:50:19.166 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.164+0000 7f75df7fe640 1 -- 172.21.15.155:0/427128302 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e8010610 con 0x7f75f806d740 2024-08-09T14:50:19.166 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.165+0000 7f75df7fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:19.166 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.165+0000 7f75f57fa640 4 set_mon_vals no callback set 2024-08-09T14:50:19.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.223+0000 7f75df7fe640 1 -- 172.21.15.155:0/427128302 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e801e060 con 0x7f75f806d740 2024-08-09T14:50:19.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.223+0000 7f75df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:19.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.223+0000 7f75df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:19.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.223+0000 7f75df7fe640 10 monclient: dump: 2024-08-09T14:50:19.225 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:19.225 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:19.225 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:19.225 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:19.225 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:19.226 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:19.226 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:19.226 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:19.226 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:19.226 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:19.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.224+0000 7f75df7fe640 1 -- 172.21.15.155:0/427128302 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f75e8017070 con 0x7f75f806d740 2024-08-09T14:50:19.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.225+0000 7f75df7fe640 1 --2- 172.21.15.155:0/427128302 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f75e406d730 0x7f75e406fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:19.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.225+0000 7f75f67fc640 1 --2- 172.21.15.155:0/427128302 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f75e406d730 0x7f75e406fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:19.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.225+0000 7f75f67fc640 10 monclient: get_auth_request con 0x7f75e406d730 auth_method 0 2024-08-09T14:50:19.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.225+0000 7f75df7fe640 10 client.4198.objecter ms_handle_connect 0x7f75e406d730 2024-08-09T14:50:19.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.225+0000 7f75f67fc640 1 --2- 172.21.15.155:0/427128302 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f75e406d730 0x7f75e406fbf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f75ec036000 tx=0x7f75ec03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:19.229 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.227+0000 7f75df7fe640 1 -- 172.21.15.155:0/427128302 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f75e80891e0 con 0x7f75f806d740 2024-08-09T14:50:19.229 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.227+0000 7f75df7fe640 10 client.4198.objecter ms_dispatch 0x7f75f8000b90 osd_map(10..10 src has 1..10) 2024-08-09T14:50:19.229 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.227+0000 7f75df7fe640 3 client.4198.objecter handle_osd_map got epochs [10,10] > 0 2024-08-09T14:50:19.229 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.227+0000 7f75df7fe640 3 client.4198.objecter handle_osd_map decoding full epoch 10 2024-08-09T14:50:19.229 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.227+0000 7f75df7fe640 20 client.4198.objecter dump_active .. 0 homeless 2024-08-09T14:50:19.239 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.238+0000 7f75df7fe640 1 -- 172.21.15.155:0/427128302 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f75e8058fe0 con 0x7f75f806d740 2024-08-09T14:50:19.239 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.238+0000 7f75df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:19.239 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.238+0000 7f75df7fe640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:19.592 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.591+0000 7f75fd330640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:19.593 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.591+0000 7f75fd330640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:19.593 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.591+0000 7f75fd330640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:19.593 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.591+0000 7f75fd330640 1 -- 172.21.15.155:0/427128302 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f75f806e840 con 0x7f75f806d740 2024-08-09T14:50:19.610 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.609+0000 7f75df7fe640 1 -- 172.21.15.155:0/427128302 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f75e805d2e0 con 0x7f75f806d740 2024-08-09T14:50:19.610 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.609+0000 7f75df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:19.611 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.609+0000 7f75df7fe640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:19.611 INFO:teuthology.misc.health.smithi155.stdout: 2024-08-09T14:50:19.611 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":10,"fsid":"f3d21f4d-627f-4083-bf15-ae5f3fe632de","created":"2024-08-09T14:49:41.321598+0000","modified":"2024-08-09T14:49:46.978521+0000","last_up_change":"0.000000","last_in_change":"2024-08-09T14:49:46.978521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6526d474-661b-48e6-bcc7-6288c7477f78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"87673c5a-b124-419e-98da-17a7b7fbdf25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a14a708a-903f-4de2-8228-79e79ed135e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8e139d50-e25c-4428-9edf-c696189cb40e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0331386-2f50-4bf9-b72e-24c80f9fe9a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0f901e55-2ba8-4144-9802-3a109884254c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4b8215-814e-4b5a-9877-0ab63818b0ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"26ed84b6-639d-4ccd-bb1d-c64b593ada7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-09T14:50:19.614 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 10 librados: watch_flush enter 2024-08-09T14:50:19.614 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 10 librados: watch_flush exit 2024-08-09T14:50:19.614 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 20 client.4198.objecter shutdown clearing up homeless session... 2024-08-09T14:50:19.614 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 10 client.4198.objecter successfully canceled tick 2024-08-09T14:50:19.614 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 1 -- 172.21.15.155:0/427128302 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f75e406d730 msgr2=0x7f75e406fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:19.614 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 1 --2- 172.21.15.155:0/427128302 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f75e406d730 0x7f75e406fbf0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f75ec036000 tx=0x7f75ec03d040 comp rx=0 tx=0).stop 2024-08-09T14:50:19.614 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 10 monclient: shutdown 2024-08-09T14:50:19.615 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:19.615 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 1 -- 172.21.15.155:0/427128302 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75f806d740 msgr2=0x7f75f8070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:19.615 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 1 --2- 172.21.15.155:0/427128302 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75f806d740 0x7f75f8070140 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f75e800b4a0 tx=0x7f75e800b960 comp rx=0 tx=0).stop 2024-08-09T14:50:19.615 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 1 -- 172.21.15.155:0/427128302 shutdown_connections 2024-08-09T14:50:19.615 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 1 --2- 172.21.15.155:0/427128302 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f75e406d730 0x7f75e406fbf0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:19.615 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 1 --2- 172.21.15.155:0/427128302 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f75f813cc20 0x7f75f81363d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:19.615 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 1 --2- 172.21.15.155:0/427128302 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f75f80b4a90 0x7f75f8135e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:19.615 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 1 --2- 172.21.15.155:0/427128302 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75f806d740 0x7f75f8070140 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f75e800b4a0 tx=0x7f75e800b960 comp rx=0 tx=0).stop 2024-08-09T14:50:19.615 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 1 -- 172.21.15.155:0/427128302 >> 172.21.15.155:0/427128302 conn(0x7f75f80b8d30 msgr2=0x7f75f80b5b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:19.615 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.612+0000 7f75fd330640 1 -- 172.21.15.155:0/427128302 shutdown_connections 2024-08-09T14:50:19.616 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.614+0000 7f75fd330640 1 -- 172.21.15.155:0/427128302 wait complete. 2024-08-09T14:50:19.616 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:19.614+0000 7f75fd330640 1 librados: shutdown 2024-08-09T14:50:19.625 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-08-09T14:50:23.100 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:50:23.091+0000 5dce500 -1 osd.3 0 log_to_monitors true 2024-08-09T14:50:23.234 INFO:tasks.ceph.osd.6.smithi194.stderr:2024-08-09T14:50:23.227+0000 5dce500 -1 osd.6 0 log_to_monitors true 2024-08-09T14:50:23.281 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:50:23.273+0000 5dce500 -1 osd.4 0 log_to_monitors true 2024-08-09T14:50:23.572 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:50:23.564+0000 5dce500 -1 osd.2 0 log_to_monitors true 2024-08-09T14:50:23.795 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:50:23.788+0000 5dce500 -1 osd.7 0 log_to_monitors true 2024-08-09T14:50:24.177 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:50:24.162+0000 5dce500 -1 osd.5 0 log_to_monitors true 2024-08-09T14:50:24.240 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:50:24.231+0000 5dce500 -1 osd.1 0 log_to_monitors true 2024-08-09T14:50:24.723 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:50:24.716+0000 5dce500 -1 osd.0 0 log_to_monitors true 2024-08-09T14:50:26.433 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-09T14:50:26.538 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.537+0000 7fcffc2fd640 10 monclient: get_monmap_and_config 2024-08-09T14:50:26.539 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.537+0000 7fcffc2fd640 10 monclient: build_initial_monmap 2024-08-09T14:50:26.539 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.537+0000 7fcffc2fd640 10 monclient: monmap: 2024-08-09T14:50:26.539 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:50:26.539 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:26.539 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:50:26.538642+0000 2024-08-09T14:50:26.539 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:50:26.538642+0000 2024-08-09T14:50:26.539 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:26.539 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:26.539 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:26.539 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:26.539 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:26.540 INFO:teuthology.misc.health.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:26.540 INFO:teuthology.misc.health.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:26.540 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:26.541 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.538+0000 7fcffc2fd640 1 Processor -- start 2024-08-09T14:50:26.541 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.538+0000 7fcffc2fd640 1 -- start start 2024-08-09T14:50:26.541 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.538+0000 7fcffc2fd640 10 monclient: init 2024-08-09T14:50:26.541 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.539+0000 7fcffc2fd640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:26.541 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.539+0000 7fcffc2fd640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-08-09T14:50:26.541 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.539+0000 7fcffc2fd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fcff406e3e0 0x7fcff4142c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:26.541 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.539+0000 7fcffc2fd640 10 monclient(hunting): picked mon.noname-a con 0x7fcff406e3e0 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:26.541 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.539+0000 7fcffc2fd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcff40b23d0 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:26.541 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.539+0000 7fcffc2fd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fcff40aaa90 0x7fcff40b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:26.541 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.539+0000 7fcffc2fd640 10 monclient(hunting): picked mon.noname-c con 0x7fcff40aaa90 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:26.541 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.539+0000 7fcffc2fd640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcff40b2e70 con 0x7fcff40b23d0 2024-08-09T14:50:26.542 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.539+0000 7fcffc2fd640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:26.542 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.539+0000 7fcffc2fd640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7fcff406e7c0 con 0x7fcff406e3e0 2024-08-09T14:50:26.542 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.539+0000 7fcffc2fd640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:26.542 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.539+0000 7fcffc2fd640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7fcff40b4420 con 0x7fcff40aaa90 2024-08-09T14:50:26.542 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.539+0000 7fcffc2fd640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:26.542 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.539+0000 7fcffc2fd640 10 monclient(hunting): authenticate will time out at 1233.576151s 2024-08-09T14:50:26.579 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.578+0000 7fcffa89a640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fcff40aaa90 0x7fcff40b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:26.580 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.578+0000 7fcffa099640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fcff406e3e0 0x7fcff4142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:26.581 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.580+0000 7fcffa89a640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fcff40aaa90 0x7fcff40b1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:36356/0 (socket says 172.21.15.155:36356) 2024-08-09T14:50:26.581 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.580+0000 7fcffa89a640 1 -- 172.21.15.155:0/3682338434 learned_addr learned my addr 172.21.15.155:0/3682338434 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:26.582 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.580+0000 7fcffa89a640 10 monclient(hunting): get_auth_request con 0x7fcff40aaa90 auth_method 0 2024-08-09T14:50:26.582 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.580+0000 7fcffa89a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:26.582 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.580+0000 7fcffa89a640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:26.582 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.580+0000 7fcffa89a640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:26.582 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.581+0000 7fcffa89a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:26.583 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.581+0000 7fcffa89a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:26.583 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.581+0000 7fcffa89a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:26.586 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.585+0000 7fcffa099640 10 monclient(hunting): get_auth_request con 0x7fcff406e3e0 auth_method 0 2024-08-09T14:50:26.586 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.585+0000 7fcffa099640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:26.587 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.585+0000 7fcffa099640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:26.587 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.585+0000 7fcffa099640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:26.598 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.596+0000 7fcffa89a640 10 monclient(hunting): handle_auth_done global_id 4219 payload 274 2024-08-09T14:50:26.598 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.596+0000 7fcffa89a640 1 -- 172.21.15.155:0/3682338434 >> v2:172.21.15.155:3300/0 conn(0x7fcff406e3e0 msgr2=0x7fcff4142c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:26.598 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.596+0000 7fcffa89a640 1 --2- 172.21.15.155:0/3682338434 >> v2:172.21.15.155:3300/0 conn(0x7fcff406e3e0 0x7fcff4142c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:26.598 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.596+0000 7fcffa89a640 1 -- 172.21.15.155:0/3682338434 >> v1:172.21.15.155:6789/0 conn(0x7fcff40b23d0 legacy=0x7fcff40b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:26.598 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.596+0000 7fcffa89a640 10 monclient: _finish_hunting 0 2024-08-09T14:50:26.598 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.596+0000 7fcffa89a640 1 monclient: found mon.noname-c 2024-08-09T14:50:26.598 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.596+0000 7fcffa89a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:26.598 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.596+0000 7fcffa89a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:50:26.598 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.596+0000 7fcffa89a640 1 -- 172.21.15.155:0/3682338434 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff40b1f90 con 0x7fcff40aaa90 2024-08-09T14:50:26.601 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.599+0000 7fcffa89a640 1 --2- 172.21.15.155:0/3682338434 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff40aaa90 0x7fcff40b1a50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcff4147220 tx=0x7fcfec02be50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df352aebba58a1aa server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:26.624 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.622+0000 7fcff9097640 1 -- 172.21.15.155:0/3682338434 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec02c9f0 con 0x7fcff40aaa90 2024-08-09T14:50:26.624 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.622+0000 7fcff9097640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:26.624 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.622+0000 7fcff9097640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:26.624 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.622+0000 7fcff9097640 10 monclient: dump: 2024-08-09T14:50:26.624 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:26.624 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:26.624 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:26.625 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:26.625 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:26.625 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:26.625 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:26.625 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:26.625 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:26.625 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:26.625 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcff9097640 10 monclient: _finish_auth 0 2024-08-09T14:50:26.625 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcff9097640 10 monclient: _check_auth_tickets 2024-08-09T14:50:26.625 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcff9097640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:26.625 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcff9097640 1 -- 172.21.15.155:0/3682338434 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfec02cb90 con 0x7fcff40aaa90 2024-08-09T14:50:26.625 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcff9097640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:26.625 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcff9097640 1 -- 172.21.15.155:0/3682338434 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec0343f0 con 0x7fcff40aaa90 2024-08-09T14:50:26.625 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcffc2fd640 5 monclient: authenticate success, global_id 4219 2024-08-09T14:50:26.626 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcffc2fd640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:26.626 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcff9097640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:26.626 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcffc2fd640 4 set_mon_vals no callback set 2024-08-09T14:50:26.626 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcff9097640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:26.626 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcff9097640 10 monclient: dump: 2024-08-09T14:50:26.626 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:26.626 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:26.626 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:26.626 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:26.626 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:26.626 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:26.626 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:26.626 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:26.627 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:26.627 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:26.629 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcffc2fd640 10 monclient: shutdown 2024-08-09T14:50:26.629 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcffc2fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:26.629 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3682338434 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff40aaa90 msgr2=0x7fcff40b1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:26.629 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcffc2fd640 1 --2- 172.21.15.155:0/3682338434 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff40aaa90 0x7fcff40b1a50 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcff4147220 tx=0x7fcfec02be50 comp rx=0 tx=0).stop 2024-08-09T14:50:26.629 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3682338434 shutdown_connections 2024-08-09T14:50:26.629 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcffc2fd640 1 --2- 172.21.15.155:0/3682338434 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff40aaa90 0x7fcff40b1a50 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:26.629 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcffc2fd640 1 --2- 172.21.15.155:0/3682338434 >> v2:172.21.15.155:3300/0 conn(0x7fcff406e3e0 0x7fcff4142c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:26.629 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3682338434 >> 172.21.15.155:0/3682338434 conn(0x7fcff40b52d0 msgr2=0x7fcff40b56e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:26.629 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.623+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3682338434 shutdown_connections 2024-08-09T14:50:26.630 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.624+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3682338434 wait complete. 2024-08-09T14:50:26.630 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.624+0000 7fcffc2fd640 10 monclient: build_initial_monmap 2024-08-09T14:50:26.630 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.624+0000 7fcffc2fd640 10 monclient: monmap: 2024-08-09T14:50:26.630 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:50:26.630 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:26.630 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:26.630 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-08-09T14:50:26.630 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:26.630 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:26.630 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:26.630 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:26.630 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:26.630 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 1 librados: starting msgr at 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 1 librados: starting objecter 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 1 Processor -- start 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 1 -- start start 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 1 librados: setting wanted keys 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 1 librados: calling monclient init 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 10 monclient: init 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 10 monclient: _add_conns ranks=[1,0,2] 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff406e3e0 0x7fcff413f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 10 monclient(hunting): picked mon.noname-b con 0x7fcff406e3e0 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcff40aaa90 0x7fcff413f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 10 monclient(hunting): picked mon.noname-a con 0x7fcff40aaa90 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcff41db940 0x7fcff41dbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:26.631 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 10 monclient(hunting): picked mon.noname-c con 0x7fcff41db940 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:26.632 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:26.632 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fcff414d6c0 con 0x7fcff40aaa90 2024-08-09T14:50:26.632 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:26.632 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcff40b4fc0 con 0x7fcff406e3e0 2024-08-09T14:50:26.632 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:26.632 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fcff414d840 con 0x7fcff41db940 2024-08-09T14:50:26.632 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:26.632 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.625+0000 7fcffc2fd640 10 monclient(hunting): authenticate will time out at 1233.662860s 2024-08-09T14:50:26.632 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.626+0000 7fcffa89a640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcff41db940 0x7fcff41dbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:26.632 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.626+0000 7fcffa89a640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcff41db940 0x7fcff41dbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:41962/0 (socket says 172.21.15.155:41962) 2024-08-09T14:50:26.632 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.626+0000 7fcffa89a640 1 -- 172.21.15.155:0/3407859274 learned_addr learned my addr 172.21.15.155:0/3407859274 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:26.632 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.626+0000 7fcffa89a640 10 monclient(hunting): get_auth_request con 0x7fcff41db940 auth_method 0 2024-08-09T14:50:26.632 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.626+0000 7fcffa89a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:26.633 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.626+0000 7fcffa89a640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:26.633 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.626+0000 7fcffa89a640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:26.633 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.627+0000 7fcffa89a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:26.633 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.627+0000 7fcffa89a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:26.633 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.627+0000 7fcffa89a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:26.634 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.630+0000 7fcffa89a640 10 monclient(hunting): handle_auth_done global_id 4220 payload 824 2024-08-09T14:50:26.634 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.630+0000 7fcffa89a640 1 -- 172.21.15.155:0/3407859274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff406e3e0 msgr2=0x7fcff413f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:26.634 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.630+0000 7fcffa89a640 1 --2- 172.21.15.155:0/3407859274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff406e3e0 0x7fcff413f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:26.634 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.630+0000 7fcffa89a640 1 -- 172.21.15.155:0/3407859274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcff40aaa90 msgr2=0x7fcff413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:26.634 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.630+0000 7fcffa89a640 1 --2- 172.21.15.155:0/3407859274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcff40aaa90 0x7fcff413f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:26.634 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.630+0000 7fcffa89a640 10 monclient: _finish_hunting 0 2024-08-09T14:50:26.634 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.630+0000 7fcffa89a640 1 monclient: found mon.noname-c 2024-08-09T14:50:26.634 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.630+0000 7fcffa89a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:26.634 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.630+0000 7fcffa89a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:50:26.634 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.630+0000 7fcffa89a640 1 -- 172.21.15.155:0/3407859274 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff4144c70 con 0x7fcff41db940 2024-08-09T14:50:26.635 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.633+0000 7fcffa89a640 1 --2- 172.21.15.155:0/3407859274 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcff41db940 0x7fcff41dbda0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcfec036040 tx=0x7fcfec004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:26.635 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.633+0000 7fcfeaffd640 10 client.?.objecter ms_handle_connect 0x7fcff41db940 2024-08-09T14:50:26.635 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.633+0000 7fcfeaffd640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:26.637 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.635+0000 7fcfeaffd640 1 -- 172.21.15.155:0/3407859274 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec043020 con 0x7fcff41db940 2024-08-09T14:50:26.637 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.635+0000 7fcfeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:26.637 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.635+0000 7fcfeaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:26.637 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.635+0000 7fcfeaffd640 10 monclient: dump: 2024-08-09T14:50:26.637 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:26.637 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:26.637 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:26.638 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:26.638 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:26.638 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:26.638 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:26.638 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:26.638 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:26.638 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:26.638 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.635+0000 7fcfeaffd640 10 monclient: _finish_auth 0 2024-08-09T14:50:26.638 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.635+0000 7fcfeaffd640 10 monclient: _check_auth_tickets 2024-08-09T14:50:26.638 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.635+0000 7fcfeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:26.638 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.635+0000 7fcffc2fd640 5 monclient: authenticate success, global_id 4220 2024-08-09T14:50:26.638 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.635+0000 7fcffc2fd640 10 monclient: _renew_subs 2024-08-09T14:50:26.638 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.635+0000 7fcffc2fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:26.639 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.635+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3407859274 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff4144f50 con 0x7fcff41db940 2024-08-09T14:50:26.639 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.636+0000 7fcffc2fd640 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:26.639 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.636+0000 7fcffc2fd640 10 monclient: _renew_subs 2024-08-09T14:50:26.639 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.636+0000 7fcffc2fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:26.639 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.636+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3407859274 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff41dc2e0 con 0x7fcff41db940 2024-08-09T14:50:26.639 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.636+0000 7fcffc2fd640 1 librados: init done 2024-08-09T14:50:26.640 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.636+0000 7fcffc2fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:26.640 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.636+0000 7fcffc2fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:26.640 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.636+0000 7fcffc2fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:26.640 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.636+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3407859274 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcff4142c80 con 0x7fcff41db940 2024-08-09T14:50:26.644 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:50:26.640+0000 c49a640 -1 osd.3 0 waiting for initial osdmap 2024-08-09T14:50:26.733 INFO:tasks.ceph.osd.6.smithi194.stderr:2024-08-09T14:50:26.728+0000 ac84640 -1 osd.6 0 waiting for initial osdmap 2024-08-09T14:50:26.784 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:50:26.781+0000 c49a640 -1 osd.4 0 waiting for initial osdmap 2024-08-09T14:50:26.937 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.936+0000 7fcfeaffd640 1 -- 172.21.15.155:0/3407859274 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcfec02ca00 con 0x7fcff41db940 2024-08-09T14:50:26.938 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.936+0000 7fcfeaffd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:26.938 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.936+0000 7fcfeaffd640 1 -- 172.21.15.155:0/3407859274 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfec009510 con 0x7fcff41db940 2024-08-09T14:50:26.938 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.936+0000 7fcfeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:26.938 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.936+0000 7fcfeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:26.938 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.936+0000 7fcfeaffd640 10 monclient: dump: 2024-08-09T14:50:26.938 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:26.938 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:26.938 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:26.938 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:26.938 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:26.938 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:26.939 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:26.939 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:26.939 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:26.939 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:26.939 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.936+0000 7fcff8896640 4 set_mon_vals no callback set 2024-08-09T14:50:26.940 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.937+0000 7fcfeaffd640 1 -- 172.21.15.155:0/3407859274 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7fcfec046020 con 0x7fcff41db940 2024-08-09T14:50:26.940 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.938+0000 7fcfeaffd640 1 --2- 172.21.15.155:0/3407859274 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fcfd806d760 0x7fcfd806fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:26.940 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.938+0000 7fcffa099640 1 --2- 172.21.15.155:0/3407859274 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fcfd806d760 0x7fcfd806fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:26.940 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.938+0000 7fcffa099640 10 monclient: get_auth_request con 0x7fcfd806d760 auth_method 0 2024-08-09T14:50:26.940 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.938+0000 7fcfeaffd640 10 client.4220.objecter ms_handle_connect 0x7fcfd806d760 2024-08-09T14:50:26.940 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.938+0000 7fcffa099640 1 --2- 172.21.15.155:0/3407859274 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fcfd806d760 0x7fcfd806fc20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcff00098f0 tx=0x7fcff0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:26.954 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.953+0000 7fcfeaffd640 1 -- 172.21.15.155:0/3407859274 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(12..12 src has 1..12) ==== 2183+0+0 (secure 0 0 0) 0x7fcfec0ac060 con 0x7fcff41db940 2024-08-09T14:50:26.954 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.953+0000 7fcfeaffd640 10 client.4220.objecter ms_dispatch 0x7fcff4000b90 osd_map(12..12 src has 1..12) 2024-08-09T14:50:26.954 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.953+0000 7fcfeaffd640 3 client.4220.objecter handle_osd_map got epochs [12,12] > 0 2024-08-09T14:50:26.954 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.953+0000 7fcfeaffd640 3 client.4220.objecter handle_osd_map decoding full epoch 12 2024-08-09T14:50:26.955 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.953+0000 7fcfeaffd640 20 client.4220.objecter dump_active .. 0 homeless 2024-08-09T14:50:26.955 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.953+0000 7fcfeaffd640 1 -- 172.21.15.155:0/3407859274 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7fcfec04d030 con 0x7fcff41db940 2024-08-09T14:50:26.955 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.953+0000 7fcfeaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:26.955 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:26.953+0000 7fcfeaffd640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:27.089 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:50:27.073+0000 10260640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-09T14:50:27.155 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.154+0000 7fcffc2fd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:27.155 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.154+0000 7fcffc2fd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:27.155 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.154+0000 7fcffc2fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:27.155 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.154+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3407859274 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcff406e9b0 con 0x7fcff41db940 2024-08-09T14:50:27.160 INFO:tasks.ceph.osd.6.smithi194.stderr:2024-08-09T14:50:27.145+0000 10260640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-09T14:50:27.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.179+0000 7fcfeaffd640 1 -- 172.21.15.155:0/3407859274 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7fcfec07c4b0 con 0x7fcff41db940 2024-08-09T14:50:27.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.179+0000 7fcfeaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:27.180 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.179+0000 7fcfeaffd640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:27.181 INFO:teuthology.misc.health.smithi155.stdout: 2024-08-09T14:50:27.181 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":12,"fsid":"f3d21f4d-627f-4083-bf15-ae5f3fe632de","created":"2024-08-09T14:49:41.321598+0000","modified":"2024-08-09T14:50:26.229097+0000","last_up_change":"0.000000","last_in_change":"2024-08-09T14:49:46.978521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"6526d474-661b-48e6-bcc7-6288c7477f78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"87673c5a-b124-419e-98da-17a7b7fbdf25","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a14a708a-903f-4de2-8228-79e79ed135e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8e139d50-e25c-4428-9edf-c696189cb40e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0331386-2f50-4bf9-b72e-24c80f9fe9a8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"0f901e55-2ba8-4144-9802-3a109884254c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fd4b8215-814e-4b5a-9877-0ab63818b0ba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"26ed84b6-639d-4ccd-bb1d-c64b593ada7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-09T14:50:27.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.181+0000 7fcffc2fd640 10 librados: watch_flush enter 2024-08-09T14:50:27.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.181+0000 7fcffc2fd640 10 librados: watch_flush exit 2024-08-09T14:50:27.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.181+0000 7fcffc2fd640 20 client.4220.objecter shutdown clearing up homeless session... 2024-08-09T14:50:27.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.181+0000 7fcffc2fd640 10 client.4220.objecter successfully canceled tick 2024-08-09T14:50:27.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.181+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3407859274 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fcfd806d760 msgr2=0x7fcfd806fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:27.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.181+0000 7fcffc2fd640 1 --2- 172.21.15.155:0/3407859274 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fcfd806d760 0x7fcfd806fc20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fcff00098f0 tx=0x7fcff0008040 comp rx=0 tx=0).stop 2024-08-09T14:50:27.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.181+0000 7fcffc2fd640 10 monclient: shutdown 2024-08-09T14:50:27.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.181+0000 7fcffc2fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:27.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.181+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3407859274 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcff41db940 msgr2=0x7fcff41dbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:27.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.181+0000 7fcffc2fd640 1 --2- 172.21.15.155:0/3407859274 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcff41db940 0x7fcff41dbda0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcfec036040 tx=0x7fcfec004a40 comp rx=0 tx=0).stop 2024-08-09T14:50:27.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.182+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3407859274 shutdown_connections 2024-08-09T14:50:27.188 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.182+0000 7fcffc2fd640 1 --2- 172.21.15.155:0/3407859274 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fcfd806d760 0x7fcfd806fc20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:27.189 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.182+0000 7fcffc2fd640 1 --2- 172.21.15.155:0/3407859274 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fcff41db940 0x7fcff41dbda0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:27.189 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.182+0000 7fcffc2fd640 1 --2- 172.21.15.155:0/3407859274 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fcff40aaa90 0x7fcff413f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:27.189 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.182+0000 7fcffc2fd640 1 --2- 172.21.15.155:0/3407859274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcff406e3e0 0x7fcff413f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:27.189 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.182+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3407859274 >> 172.21.15.155:0/3407859274 conn(0x7fcff40b52d0 msgr2=0x7fcff40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:27.189 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.182+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3407859274 shutdown_connections 2024-08-09T14:50:27.189 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.182+0000 7fcffc2fd640 1 -- 172.21.15.155:0/3407859274 wait complete. 2024-08-09T14:50:27.189 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:27.182+0000 7fcffc2fd640 1 librados: shutdown 2024-08-09T14:50:27.195 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-08-09T14:50:27.213 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:50:27.198+0000 10260640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-09T14:50:27.828 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:50:27.823+0000 c49a640 -1 osd.5 0 waiting for initial osdmap 2024-08-09T14:50:27.880 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:50:27.873+0000 c49a640 -1 osd.1 0 waiting for initial osdmap 2024-08-09T14:50:28.211 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:50:28.196+0000 10260640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-09T14:50:28.292 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:50:28.277+0000 10260640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-09T14:50:33.985 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:50:33.604+0000 5dce500 -1 bdev(0x164bc580 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-08-09T14:50:34.001 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-09T14:50:34.108 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.107+0000 7f9abeb25640 10 monclient: get_monmap_and_config 2024-08-09T14:50:34.109 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.107+0000 7f9abeb25640 10 monclient: build_initial_monmap 2024-08-09T14:50:34.109 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.107+0000 7f9abeb25640 10 monclient: monmap: 2024-08-09T14:50:34.109 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:50:34.109 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:34.109 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:50:34.108591+0000 2024-08-09T14:50:34.109 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:50:34.108591+0000 2024-08-09T14:50:34.109 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:34.109 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:34.109 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:34.109 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:34.109 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:34.109 INFO:teuthology.misc.health.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:34.109 INFO:teuthology.misc.health.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:34.109 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:34.111 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.108+0000 7f9abeb25640 1 Processor -- start 2024-08-09T14:50:34.111 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.108+0000 7f9abeb25640 1 -- start start 2024-08-09T14:50:34.111 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.108+0000 7f9abeb25640 10 monclient: init 2024-08-09T14:50:34.111 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.109+0000 7f9abeb25640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:34.111 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.109+0000 7f9abeb25640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-08-09T14:50:34.111 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.109+0000 7f9abeb25640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ab806d740 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:34.111 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.109+0000 7f9abeb25640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ab80b4550 0x7f9ab80b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:34.112 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.109+0000 7f9abeb25640 10 monclient(hunting): picked mon.noname-b con 0x7f9ab80b4550 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:34.112 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.109+0000 7f9abeb25640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ab8142ce0 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:34.112 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.109+0000 7f9abeb25640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab8143880 con 0x7f9ab8142ce0 2024-08-09T14:50:34.112 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.109+0000 7f9abeb25640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab813cdf0 con 0x7f9ab806d740 2024-08-09T14:50:34.112 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.109+0000 7f9abeb25640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:34.112 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.109+0000 7f9abeb25640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f9ab806db20 con 0x7f9ab80b4550 2024-08-09T14:50:34.112 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.109+0000 7f9abeb25640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:34.112 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.109+0000 7f9abeb25640 10 monclient(hunting): authenticate will time out at 1241.146324s 2024-08-09T14:50:34.112 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.109+0000 7f9abc8c1640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f9ab806d740 0x7f9ab813c6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.155:55460/0 (socket says 172.21.15.155:55460) 2024-08-09T14:50:34.112 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.109+0000 7f9abc8c1640 1 -- 172.21.15.155:0/2199650844 learned_addr learned my addr 172.21.15.155:0/2199650844 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:34.114 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.113+0000 7f9ab77fe640 1 -- 172.21.15.155:0/2199650844 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1470552206 0 0) 0x7f9ab8143880 con 0x7f9ab8142ce0 2024-08-09T14:50:34.114 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.113+0000 7f9ab77fe640 10 monclient(hunting): _negotiate 2024-08-09T14:50:34.115 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.113+0000 7f9ab77fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:34.115 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.113+0000 7f9ab77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:34.115 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.113+0000 7f9ab77fe640 10 monclient(hunting): my global_id is 4236 2024-08-09T14:50:34.115 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.113+0000 7f9ab77fe640 1 -- 172.21.15.155:0/2199650844 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9aa8003520 con 0x7f9ab8142ce0 2024-08-09T14:50:34.124 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 -- 172.21.15.155:0/2199650844 <== mon.0 v1:172.21.15.155:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 75986960 0 0) 0x7f9aa8003520 con 0x7f9ab8142ce0 2024-08-09T14:50:34.125 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 -- 172.21.15.155:0/2199650844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ab80b4550 msgr2=0x7f9ab80b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:34.125 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 --2- 172.21.15.155:0/2199650844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ab80b4550 0x7f9ab80b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:34.125 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 -- 172.21.15.155:0/2199650844 >> v1:172.21.15.194:6789/0 conn(0x7f9ab806d740 legacy=0x7f9ab813c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:34.125 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient: _finish_hunting 0 2024-08-09T14:50:34.125 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 monclient: found mon.noname-a-legacy 2024-08-09T14:50:34.125 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:34.125 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.155:6789/0 2024-08-09T14:50:34.125 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 -- 172.21.15.155:0/2199650844 --> v1:172.21.15.155:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab8150470 con 0x7f9ab8142ce0 2024-08-09T14:50:34.125 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient: _finish_auth 0 2024-08-09T14:50:34.125 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:34.125 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:34.125 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 -- 172.21.15.155:0/2199650844 <== mon.0 v1:172.21.15.155:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 1061800462 0 0) 0x7f9ab00019f0 con 0x7f9ab8142ce0 2024-08-09T14:50:34.125 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:34.126 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-08-09T14:50:34.126 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient: dump: 2024-08-09T14:50:34.126 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:34.126 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:34.126 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:34.126 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:34.126 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:34.126 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:34.126 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:34.126 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:34.126 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:34.127 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:34.127 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] but i'm connected to v1:172.21.15.155:6789/0, reconnecting 2024-08-09T14:50:34.127 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:34.127 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 -- 172.21.15.155:0/2199650844 >> v1:172.21.15.155:6789/0 conn(0x7f9ab8142ce0 legacy=0x7f9ab8143170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:34.127 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-08-09T14:50:34.127 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 --2- 172.21.15.155:0/2199650844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9aa80039e0 0x7f9aa8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:34.127 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient(hunting): picked mon.a con 0x7f9aa80039e0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:34.127 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 --2- 172.21.15.155:0/2199650844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ab80b4550 0x7f9aa8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:34.127 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient(hunting): picked mon.c con 0x7f9ab80b4550 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:34.127 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 --2- 172.21.15.155:0/2199650844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa8004e60 0x7f9aa800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:34.127 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient(hunting): picked mon.b con 0x7f9aa8004e60 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:34.127 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:34.128 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 -- 172.21.15.155:0/2199650844 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab806db20 con 0x7f9aa80039e0 2024-08-09T14:50:34.128 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:34.128 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 -- 172.21.15.155:0/2199650844 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa80055b0 con 0x7f9aa8004e60 2024-08-09T14:50:34.128 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:34.128 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 1 -- 172.21.15.155:0/2199650844 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa800bf00 con 0x7f9ab80b4550 2024-08-09T14:50:34.128 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:34.128 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.119+0000 7f9ab77fe640 10 monclient(hunting): _finish_auth 0 2024-08-09T14:50:34.128 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.121+0000 7f9abd0c2640 1 --2- 172.21.15.155:0/2199650844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa8004e60 0x7f9aa800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:34.128 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.121+0000 7f9abd0c2640 10 monclient(hunting): get_auth_request con 0x7f9aa8004e60 auth_method 0 2024-08-09T14:50:34.128 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.121+0000 7f9abd0c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:34.128 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.121+0000 7f9abd0c2640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:34.128 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.121+0000 7f9abd0c2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:50:34.128 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.122+0000 7f9abd0c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:34.129 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.122+0000 7f9abd0c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:34.129 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.122+0000 7f9abd0c2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-08-09T14:50:34.129 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.122+0000 7f9abc8c1640 1 --2- 172.21.15.155:0/2199650844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9aa80039e0 0x7f9aa8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:34.129 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.122+0000 7f9abc8c1640 10 monclient(hunting): get_auth_request con 0x7f9aa80039e0 auth_method 0 2024-08-09T14:50:34.129 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.122+0000 7f9abc8c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:34.129 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.122+0000 7f9abc8c1640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:34.129 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.122+0000 7f9abc8c1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-08-09T14:50:34.129 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.123+0000 7f9abc8c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:34.129 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.123+0000 7f9abc8c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:34.129 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.123+0000 7f9abc8c1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-08-09T14:50:34.131 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abd0c2640 10 monclient(hunting): handle_auth_done global_id 4236 payload 293 2024-08-09T14:50:34.131 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abd0c2640 1 -- 172.21.15.155:0/2199650844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ab80b4550 msgr2=0x7f9aa8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:34.131 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abd0c2640 1 --2- 172.21.15.155:0/2199650844 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ab80b4550 0x7f9aa8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:34.131 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abd0c2640 1 -- 172.21.15.155:0/2199650844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9aa80039e0 msgr2=0x7f9aa8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:34.131 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abd0c2640 1 --2- 172.21.15.155:0/2199650844 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9aa80039e0 0x7f9aa8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:34.131 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abd0c2640 10 monclient: _finish_hunting 0 2024-08-09T14:50:34.131 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abd0c2640 1 monclient: found mon.b 2024-08-09T14:50:34.132 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abd0c2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:34.132 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abd0c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:34.132 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abd0c2640 1 -- 172.21.15.155:0/2199650844 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab8150470 con 0x7f9aa8004e60 2024-08-09T14:50:34.132 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abd0c2640 10 monclient: _finish_auth 0 2024-08-09T14:50:34.132 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abd0c2640 10 monclient: _check_auth_tickets 2024-08-09T14:50:34.132 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abd0c2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:34.132 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abeb25640 5 monclient: authenticate success, global_id 4236 2024-08-09T14:50:34.132 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.124+0000 7f9abeb25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:34.132 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.126+0000 7f9abd0c2640 1 --2- 172.21.15.155:0/2199650844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa8004e60 0x7f9aa800b9c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000c510 tx=0x7f9aa8005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8514b3ff901a1a68 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:34.133 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9ab77fe640 1 -- 172.21.15.155:0/2199650844 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0039070 con 0x7f9aa8004e60 2024-08-09T14:50:34.133 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9ab77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:34.133 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9ab77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:34.133 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9ab77fe640 10 monclient: dump: 2024-08-09T14:50:34.134 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:34.134 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:34.134 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:34.134 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:34.134 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:34.134 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:34.134 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:34.134 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:34.134 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:34.134 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:34.134 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9ab77fe640 1 -- 172.21.15.155:0/2199650844 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ab002cdb0 con 0x7f9aa8004e60 2024-08-09T14:50:34.134 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9ab77fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:34.135 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9ab77fe640 1 -- 172.21.15.155:0/2199650844 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab000c0e0 con 0x7f9aa8004e60 2024-08-09T14:50:34.135 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9ab77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:34.135 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9ab77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:34.135 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9ab77fe640 10 monclient: dump: 2024-08-09T14:50:34.135 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:34.135 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:34.135 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:34.135 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:34.135 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:34.135 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:34.135 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:34.135 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:34.135 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:34.136 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:34.136 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9abeb25640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:34.136 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9abeb25640 4 set_mon_vals no callback set 2024-08-09T14:50:34.136 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9abeb25640 10 monclient: shutdown 2024-08-09T14:50:34.136 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9abeb25640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:34.136 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9abeb25640 1 -- 172.21.15.155:0/2199650844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa8004e60 msgr2=0x7f9aa800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:34.136 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.131+0000 7f9abeb25640 1 --2- 172.21.15.155:0/2199650844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9aa8004e60 0x7f9aa800b9c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000c510 tx=0x7f9aa8005830 comp rx=0 tx=0).stop 2024-08-09T14:50:34.136 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.132+0000 7f9abd0c2640 1 -- 172.21.15.155:0/2199650844 reap_dead start 2024-08-09T14:50:34.136 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.132+0000 7f9abeb25640 1 -- 172.21.15.155:0/2199650844 shutdown_connections 2024-08-09T14:50:34.136 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.132+0000 7f9abeb25640 1 -- 172.21.15.155:0/2199650844 >> 172.21.15.155:0/2199650844 conn(0x7f9ab80b8d30 msgr2=0x7f9ab80b9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:34.140 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.133+0000 7f9abeb25640 1 -- 172.21.15.155:0/2199650844 shutdown_connections 2024-08-09T14:50:34.140 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.133+0000 7f9abeb25640 1 -- 172.21.15.155:0/2199650844 wait complete. 2024-08-09T14:50:34.140 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.134+0000 7f9abeb25640 10 monclient: build_initial_monmap 2024-08-09T14:50:34.140 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.134+0000 7f9abeb25640 10 monclient: monmap: 2024-08-09T14:50:34.140 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:50:34.140 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:34.140 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:34.140 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-08-09T14:50:34.141 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:34.141 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:34.141 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:34.141 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:34.141 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:34.141 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:34.141 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.134+0000 7f9abeb25640 1 librados: starting msgr at 2024-08-09T14:50:34.141 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.134+0000 7f9abeb25640 1 librados: starting objecter 2024-08-09T14:50:34.141 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.134+0000 7f9abeb25640 1 Processor -- start 2024-08-09T14:50:34.141 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.134+0000 7f9abeb25640 1 -- start start 2024-08-09T14:50:34.141 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.134+0000 7f9abeb25640 1 librados: setting wanted keys 2024-08-09T14:50:34.141 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.134+0000 7f9abeb25640 1 librados: calling monclient init 2024-08-09T14:50:34.141 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.134+0000 7f9abeb25640 10 monclient: init 2024-08-09T14:50:34.142 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.135+0000 7f9abeb25640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:34.142 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.135+0000 7f9abeb25640 10 monclient: _add_conns ranks=[0,1,2] 2024-08-09T14:50:34.142 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.135+0000 7f9abeb25640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9aa8004e60 0x7f9ab80715b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:34.142 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.135+0000 7f9abeb25640 10 monclient(hunting): picked mon.noname-a con 0x7f9aa8004e60 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:34.142 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.135+0000 7f9abeb25640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ab8071b40 0x7f9ab806ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:34.142 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.136+0000 7f9abeb25640 10 monclient(hunting): picked mon.noname-b con 0x7f9ab8071b40 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:34.142 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.136+0000 7f9abeb25640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ab806f180 0x7f9ab81d7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:34.142 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.136+0000 7f9abeb25640 10 monclient(hunting): picked mon.noname-c con 0x7f9ab806f180 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:34.142 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.136+0000 7f9abeb25640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:34.142 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.136+0000 7f9abeb25640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab80b4d00 con 0x7f9aa8004e60 2024-08-09T14:50:34.142 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.136+0000 7f9abeb25640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:34.142 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.136+0000 7f9abeb25640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab80b4b80 con 0x7f9ab8071b40 2024-08-09T14:50:34.142 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.136+0000 7f9abeb25640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:34.143 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.136+0000 7f9abeb25640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f9ab80b4e80 con 0x7f9ab806f180 2024-08-09T14:50:34.143 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.136+0000 7f9abeb25640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:34.143 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.136+0000 7f9abeb25640 10 monclient(hunting): authenticate will time out at 1241.173234s 2024-08-09T14:50:34.143 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.136+0000 7f9ab7fff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ab8071b40 0x7f9ab806ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:34.143 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.137+0000 7f9ab7fff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ab8071b40 0x7f9ab806ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:36386/0 (socket says 172.21.15.155:36386) 2024-08-09T14:50:34.143 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.137+0000 7f9ab7fff640 1 -- 172.21.15.155:0/351252827 learned_addr learned my addr 172.21.15.155:0/351252827 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:34.143 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.137+0000 7f9ab7fff640 10 monclient(hunting): get_auth_request con 0x7f9ab8071b40 auth_method 0 2024-08-09T14:50:34.143 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.137+0000 7f9ab7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:34.143 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.137+0000 7f9ab7fff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:34.143 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.137+0000 7f9ab7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:34.143 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.137+0000 7f9ab7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:34.143 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.137+0000 7f9ab7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:34.143 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.137+0000 7f9ab7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:34.143 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.139+0000 7f9abc8c1640 1 --2- 172.21.15.155:0/351252827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9aa8004e60 0x7f9ab80715b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:34.146 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.139+0000 7f9abc8c1640 10 monclient(hunting): get_auth_request con 0x7f9aa8004e60 auth_method 0 2024-08-09T14:50:34.146 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.139+0000 7f9abc8c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:34.146 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.139+0000 7f9abc8c1640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:34.146 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.139+0000 7f9abc8c1640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:34.146 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9abc8c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:34.146 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9abd0c2640 1 --2- 172.21.15.155:0/351252827 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ab806f180 0x7f9ab81d7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:34.146 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9abc8c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:34.147 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9abc8c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:34.147 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9abd0c2640 10 monclient(hunting): get_auth_request con 0x7f9ab806f180 auth_method 0 2024-08-09T14:50:34.147 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9abd0c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:34.147 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9abd0c2640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:34.147 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9abd0c2640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:34.147 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9ab7fff640 10 monclient(hunting): handle_auth_done global_id 4222 payload 824 2024-08-09T14:50:34.147 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9ab7fff640 1 -- 172.21.15.155:0/351252827 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ab806f180 msgr2=0x7f9ab81d7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:34.147 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9ab7fff640 1 --2- 172.21.15.155:0/351252827 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ab806f180 0x7f9ab81d7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:34.147 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9ab7fff640 1 -- 172.21.15.155:0/351252827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9aa8004e60 msgr2=0x7f9ab80715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:34.147 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9ab7fff640 1 --2- 172.21.15.155:0/351252827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9aa8004e60 0x7f9ab80715b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:34.147 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9ab7fff640 10 monclient: _finish_hunting 0 2024-08-09T14:50:34.147 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9ab7fff640 1 monclient: found mon.noname-b 2024-08-09T14:50:34.147 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9ab7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:34.148 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9ab7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:34.148 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.140+0000 7f9ab7fff640 1 -- 172.21.15.155:0/351252827 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab8071f80 con 0x7f9ab8071b40 2024-08-09T14:50:34.148 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.143+0000 7f9ab7fff640 1 --2- 172.21.15.155:0/351252827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ab8071b40 0x7f9ab806ec40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400b480 tx=0x7f9aa400b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:34.149 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.143+0000 7f9ab57fa640 10 client.?.objecter ms_handle_connect 0x7f9ab8071b40 2024-08-09T14:50:34.149 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.143+0000 7f9ab57fa640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:34.221 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9ab57fa640 1 -- 172.21.15.155:0/351252827 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4014070 con 0x7f9ab8071b40 2024-08-09T14:50:34.221 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9ab57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:34.221 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9ab57fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:34.221 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9ab57fa640 10 monclient: dump: 2024-08-09T14:50:34.221 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:34.222 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:34.222 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:34.222 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:34.222 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:34.222 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:34.223 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:34.223 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:34.223 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:34.223 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:34.223 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9ab57fa640 10 monclient: _finish_auth 0 2024-08-09T14:50:34.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9ab57fa640 10 monclient: _check_auth_tickets 2024-08-09T14:50:34.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9ab57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:34.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9ab57fa640 1 -- 172.21.15.155:0/351252827 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa4010870 con 0x7f9ab8071b40 2024-08-09T14:50:34.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9ab57fa640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:34.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9ab57fa640 1 -- 172.21.15.155:0/351252827 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4010c20 con 0x7f9ab8071b40 2024-08-09T14:50:34.224 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9ab57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:34.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9ab57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:34.225 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9ab57fa640 10 monclient: dump: 2024-08-09T14:50:34.225 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:34.225 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:34.225 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:34.225 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:34.226 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:34.226 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:34.226 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:34.226 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:34.226 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:34.226 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:34.226 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9abeb25640 5 monclient: authenticate success, global_id 4222 2024-08-09T14:50:34.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9abeb25640 10 monclient: _renew_subs 2024-08-09T14:50:34.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9abeb25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:34.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9abeb25640 1 -- 172.21.15.155:0/351252827 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab80acbd0 con 0x7f9ab8071b40 2024-08-09T14:50:34.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9ab6ffd640 4 set_mon_vals no callback set 2024-08-09T14:50:34.227 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9abeb25640 10 client.4222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:34.228 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9abeb25640 10 monclient: _renew_subs 2024-08-09T14:50:34.228 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9abeb25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:34.228 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9abeb25640 1 -- 172.21.15.155:0/351252827 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab8072710 con 0x7f9ab8071b40 2024-08-09T14:50:34.228 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.219+0000 7f9abeb25640 1 librados: init done 2024-08-09T14:50:34.229 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.220+0000 7f9abeb25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:34.229 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.220+0000 7f9abeb25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:34.229 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.220+0000 7f9abeb25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:34.229 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.220+0000 7f9abeb25640 1 -- 172.21.15.155:0/351252827 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab8072240 con 0x7f9ab8071b40 2024-08-09T14:50:34.410 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:50:34.252+0000 5dce500 -1 bdev(0x18ef1620 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 3 2024-08-09T14:50:34.434 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:50:34.409+0000 5dce500 -1 bdev(0x18ef1620 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-08-09T14:50:34.496 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.495+0000 7f9ab57fa640 1 -- 172.21.15.155:0/351252827 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f9aa4017070 con 0x7f9ab8071b40 2024-08-09T14:50:34.497 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.495+0000 7f9ab57fa640 1 --2- 172.21.15.155:0/351252827 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f9a9c06d780 0x7f9a9c06fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:34.497 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.496+0000 7f9abc8c1640 1 --2- 172.21.15.155:0/351252827 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f9a9c06d780 0x7f9a9c06fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:34.497 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.496+0000 7f9abc8c1640 10 monclient: get_auth_request con 0x7f9a9c06d780 auth_method 0 2024-08-09T14:50:34.497 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.496+0000 7f9ab57fa640 10 client.4222.objecter ms_handle_connect 0x7f9a9c06d780 2024-08-09T14:50:34.498 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.496+0000 7f9abc8c1640 1 --2- 172.21.15.155:0/351252827 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f9a9c06d780 0x7f9a9c06fc40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9aac000c00 tx=0x7f9aac007d70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:34.501 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.500+0000 7f9ab57fa640 1 -- 172.21.15.155:0/351252827 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 4017+0+0 (secure 0 0 0) 0x7f9aa4057380 con 0x7f9ab8071b40 2024-08-09T14:50:34.501 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.500+0000 7f9ab57fa640 10 client.4222.objecter ms_dispatch 0x7f9ab8000b90 osd_map(18..18 src has 1..18) 2024-08-09T14:50:34.501 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.500+0000 7f9ab57fa640 3 client.4222.objecter handle_osd_map got epochs [18,18] > 0 2024-08-09T14:50:34.502 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.500+0000 7f9ab57fa640 3 client.4222.objecter handle_osd_map decoding full epoch 18 2024-08-09T14:50:34.502 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.500+0000 7f9ab57fa640 20 client.4222.objecter dump_active .. 0 homeless 2024-08-09T14:50:34.512 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.511+0000 7f9ab57fa640 1 -- 172.21.15.155:0/351252827 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f9aa40592e0 con 0x7f9ab8071b40 2024-08-09T14:50:34.513 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.511+0000 7f9ab57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:34.513 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.511+0000 7f9ab57fa640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:34.562 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:50:34.461+0000 5dce500 -1 bdev(0x18ef1620 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-08-09T14:50:34.562 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:50:34.507+0000 5dce500 -1 bdev(0x18ef1620 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-08-09T14:50:34.592 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:50:34.124+0000 5dce500 -1 bdev(0x164bc580 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 1 2024-08-09T14:50:34.616 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:50:34.562+0000 5dce500 -1 bdev(0x18ef1620 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-08-09T14:50:34.635 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:50:34.593+0000 5dce500 -1 bdev(0x164bc580 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 2 2024-08-09T14:50:34.746 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.745+0000 7f9abeb25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:34.746 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.745+0000 7f9abeb25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:34.746 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.745+0000 7f9abeb25640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:34.746 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.745+0000 7f9abeb25640 1 -- 172.21.15.155:0/351252827 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9ab806f560 con 0x7f9ab8071b40 2024-08-09T14:50:34.802 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:50:34.619+0000 5dce500 -1 bdev(0x18ef1620 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-08-09T14:50:34.803 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:50:34.795+0000 5dce500 -1 bdev(0x18ef1620 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-08-09T14:50:34.843 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.841+0000 7f9ab57fa640 1 -- 172.21.15.155:0/351252827 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) ==== 74+0+11417 (secure 0 0 0) 0x7f9ab806f560 con 0x7f9ab8071b40 2024-08-09T14:50:34.843 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.841+0000 7f9ab57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:34.843 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.841+0000 7f9ab57fa640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:34.843 INFO:teuthology.misc.health.smithi155.stdout: 2024-08-09T14:50:34.843 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":18,"fsid":"f3d21f4d-627f-4083-bf15-ae5f3fe632de","created":"2024-08-09T14:49:41.321598+0000","modified":"2024-08-09T14:50:33.968115+0000","last_up_change":"2024-08-09T14:50:29.494291+0000","last_in_change":"2024-08-09T14:49:46.978521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-09T14:50:30.139923+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6526d474-661b-48e6-bcc7-6288c7477f78","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"87673c5a-b124-419e-98da-17a7b7fbdf25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6819","nonce":3129678288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6821","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6823","nonce":3129678288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6827","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6829","nonce":3129678288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6825","nonce":3129678288}]},"public_addr":"172.21.15.155:6819/3129678288","cluster_addr":"172.21.15.155:6823/3129678288","heartbeat_back_addr":"172.21.15.155:6829/3129678288","heartbeat_front_addr":"172.21.15.155:6825/3129678288","state":["exists","up"]},{"osd":2,"uuid":"a14a708a-903f-4de2-8228-79e79ed135e6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"8e139d50-e25c-4428-9edf-c696189cb40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6811","nonce":3967029756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6813","nonce":3967029756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6817","nonce":3967029756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6815","nonce":3967029756}]},"public_addr":"172.21.15.155:6811/3967029756","cluster_addr":"172.21.15.155:6813/3967029756","heartbeat_back_addr":"172.21.15.155:6817/3967029756","heartbeat_front_addr":"172.21.15.155:6815/3967029756","state":["exists","up"]},{"osd":4,"uuid":"d0331386-2f50-4bf9-b72e-24c80f9fe9a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6801","nonce":2229949751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6803","nonce":2229949751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6807","nonce":2229949751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6805","nonce":2229949751}]},"public_addr":"172.21.15.194:6801/2229949751","cluster_addr":"172.21.15.194:6803/2229949751","heartbeat_back_addr":"172.21.15.194:6807/2229949751","heartbeat_front_addr":"172.21.15.194:6805/2229949751","state":["exists","up"]},{"osd":5,"uuid":"0f901e55-2ba8-4144-9802-3a109884254c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6817","nonce":1013364343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6819","nonce":1013364343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6825","nonce":1013364343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6821","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6822","nonce":1013364343}]},"public_addr":"172.21.15.194:6817/1013364343","cluster_addr":"172.21.15.194:6819/1013364343","heartbeat_back_addr":"172.21.15.194:6825/1013364343","heartbeat_front_addr":"172.21.15.194:6822/1013364343","state":["exists","up"]},{"osd":6,"uuid":"fd4b8215-814e-4b5a-9877-0ab63818b0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6809","nonce":3077539270}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6811","nonce":3077539270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6815","nonce":3077539270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6813","nonce":3077539270}]},"public_addr":"172.21.15.194:6809/3077539270","cluster_addr":"172.21.15.194:6811/3077539270","heartbeat_back_addr":"172.21.15.194:6815/3077539270","heartbeat_front_addr":"172.21.15.194:6813/3077539270","state":["exists","up"]},{"osd":7,"uuid":"26ed84b6-639d-4ccd-bb1d-c64b593ada7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:26.207562+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.163852+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.187804+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:26.108803+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.199884+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-09T14:50:34.845 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.843+0000 7f9abeb25640 10 librados: watch_flush enter 2024-08-09T14:50:34.845 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.843+0000 7f9abeb25640 10 librados: watch_flush exit 2024-08-09T14:50:34.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.843+0000 7f9abeb25640 20 client.4222.objecter shutdown clearing up homeless session... 2024-08-09T14:50:34.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.843+0000 7f9abeb25640 10 client.4222.objecter successfully canceled tick 2024-08-09T14:50:34.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.843+0000 7f9abeb25640 1 -- 172.21.15.155:0/351252827 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f9a9c06d780 msgr2=0x7f9a9c06fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:34.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.843+0000 7f9abeb25640 1 --2- 172.21.15.155:0/351252827 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f9a9c06d780 0x7f9a9c06fc40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f9aac000c00 tx=0x7f9aac007d70 comp rx=0 tx=0).stop 2024-08-09T14:50:34.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.843+0000 7f9abeb25640 10 monclient: shutdown 2024-08-09T14:50:34.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.843+0000 7f9abeb25640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:34.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.843+0000 7f9abeb25640 1 -- 172.21.15.155:0/351252827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ab8071b40 msgr2=0x7f9ab806ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:34.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.843+0000 7f9abeb25640 1 --2- 172.21.15.155:0/351252827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ab8071b40 0x7f9ab806ec40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400b480 tx=0x7f9aa400b940 comp rx=0 tx=0).stop 2024-08-09T14:50:34.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.844+0000 7f9abeb25640 1 -- 172.21.15.155:0/351252827 shutdown_connections 2024-08-09T14:50:34.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.844+0000 7f9abeb25640 1 --2- 172.21.15.155:0/351252827 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f9a9c06d780 0x7f9a9c06fc40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:34.846 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.844+0000 7f9abeb25640 1 --2- 172.21.15.155:0/351252827 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9ab806f180 0x7f9ab81d7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:34.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.844+0000 7f9abeb25640 1 --2- 172.21.15.155:0/351252827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ab8071b40 0x7f9ab806ec40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:34.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.844+0000 7f9abeb25640 1 --2- 172.21.15.155:0/351252827 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f9aa8004e60 0x7f9ab80715b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:34.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.844+0000 7f9abeb25640 1 -- 172.21.15.155:0/351252827 >> 172.21.15.155:0/351252827 conn(0x7f9ab80b8d30 msgr2=0x7f9ab80b3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:34.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.844+0000 7f9abeb25640 1 -- 172.21.15.155:0/351252827 shutdown_connections 2024-08-09T14:50:34.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.844+0000 7f9abeb25640 1 -- 172.21.15.155:0/351252827 wait complete. 2024-08-09T14:50:34.847 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:34.844+0000 7f9abeb25640 1 librados: shutdown 2024-08-09T14:50:34.857 DEBUG:teuthology.misc:5 of 8 OSDs are up 2024-08-09T14:50:35.323 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:50:35.298+0000 5dce500 -1 bdev(0x18ef1620 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-08-09T14:50:35.506 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:50:35.365+0000 5dce500 -1 bdev(0x164d44c0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-08-09T14:50:35.506 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:50:35.380+0000 5dce500 -1 bdev(0x164d44c0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-08-09T14:50:35.671 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:50:35.666+0000 ac84640 -1 osd.7 0 waiting for initial osdmap 2024-08-09T14:50:35.747 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:50:35.508+0000 5dce500 -1 bdev(0x164d44c0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-08-09T14:50:36.095 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:50:36.080+0000 14268640 -1 osd.7 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-09T14:50:36.125 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:50:35.746+0000 5dce500 -1 bdev(0x164d44c0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-08-09T14:50:36.200 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:50:36.100+0000 5dce500 -1 bdev(0x164d44c0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-08-09T14:50:36.387 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:50:36.124+0000 5dce500 -1 bdev(0x164d44c0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-08-09T14:50:36.389 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:50:36.199+0000 5dce500 -1 bdev(0x164d44c0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-08-09T14:50:36.389 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:50:36.227+0000 5dce500 -1 bdev(0x164d44c0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 1 2024-08-09T14:50:36.391 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:50:36.359+0000 5dce500 -1 bdev(0x164d44c0 /var/lib/ceph/osd/ceph-0/block) aio_submit retries 2 2024-08-09T14:50:36.605 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:50:36.598+0000 ac84640 -1 osd.2 0 waiting for initial osdmap 2024-08-09T14:50:37.093 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:50:37.077+0000 14268640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-09T14:50:37.522 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:50:37.517+0000 c49a640 -1 osd.0 0 waiting for initial osdmap 2024-08-09T14:50:38.067 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:50:38.051+0000 14268640 -1 osd.0 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-09T14:50:39.103 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:14.346 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:39.179 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:14.422 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:39.197 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:14.441 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:39.209 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:14.456 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:39.253 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:14.496 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:39.269 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:14.516 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:39.473 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:14.717 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:39.503 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:14.747 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:39.517 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:14.764 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:41.664 DEBUG:teuthology.orchestra.run.smithi155:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-08-09T14:50:41.756 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.754+0000 7fa8731a3640 10 monclient: get_monmap_and_config 2024-08-09T14:50:41.756 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.754+0000 7fa8731a3640 10 monclient: build_initial_monmap 2024-08-09T14:50:41.756 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.754+0000 7fa8731a3640 10 monclient: monmap: 2024-08-09T14:50:41.757 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:50:41.757 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:41.757 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:50:41.755848+0000 2024-08-09T14:50:41.757 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:50:41.755848+0000 2024-08-09T14:50:41.757 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:41.757 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:41.758 INFO:teuthology.misc.health.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:41.758 INFO:teuthology.misc.health.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:41.758 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:41.758 INFO:teuthology.misc.health.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:41.758 INFO:teuthology.misc.health.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:41.759 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:41.764 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.755+0000 7fa8731a3640 1 Processor -- start 2024-08-09T14:50:41.764 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.755+0000 7fa8731a3640 1 -- start start 2024-08-09T14:50:41.764 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.755+0000 7fa8731a3640 10 monclient: init 2024-08-09T14:50:41.765 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa8731a3640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:41.765 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa8731a3640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-08-09T14:50:41.765 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa8731a3640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa86c0b0190 0x7fa86c142db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:41.765 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa8731a3640 10 monclient(hunting): picked mon.noname-a con 0x7fa86c0b0190 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:41.765 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa8731a3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa86c0acff0 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:41.765 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa8731a3640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fa86c13cf50 0x7fa86c0ad430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:41.766 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa8731a3640 10 monclient(hunting): picked mon.noname-c con 0x7fa86c13cf50 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:41.766 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa8731a3640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa86c0ae830 con 0x7fa86c0acff0 2024-08-09T14:50:41.766 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa8731a3640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:41.766 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa8731a3640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7fa86c0b0570 con 0x7fa86c0b0190 2024-08-09T14:50:41.766 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa8731a3640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:41.766 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa8731a3640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7fa86c0b0d60 con 0x7fa86c13cf50 2024-08-09T14:50:41.766 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa8731a3640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:41.767 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa8731a3640 10 monclient(hunting): authenticate will time out at 1248.793309s 2024-08-09T14:50:41.767 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa870f3f640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa86c0b0190 0x7fa86c142db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:41.767 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.756+0000 7fa871740640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fa86c13cf50 0x7fa86c0ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:41.767 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.757+0000 7fa871740640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fa86c13cf50 0x7fa86c0ad430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:44214/0 (socket says 172.21.15.155:44214) 2024-08-09T14:50:41.767 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.757+0000 7fa871740640 1 -- 172.21.15.155:0/3825470129 learned_addr learned my addr 172.21.15.155:0/3825470129 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:41.767 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.757+0000 7fa871740640 10 monclient(hunting): get_auth_request con 0x7fa86c13cf50 auth_method 0 2024-08-09T14:50:41.767 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.757+0000 7fa871740640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:41.767 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.757+0000 7fa871740640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:41.767 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.757+0000 7fa871740640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:41.768 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.757+0000 7fa870f3f640 10 monclient(hunting): get_auth_request con 0x7fa86c0b0190 auth_method 0 2024-08-09T14:50:41.768 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.757+0000 7fa870f3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:41.768 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.757+0000 7fa870f3f640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:41.768 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.757+0000 7fa870f3f640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:41.768 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.757+0000 7fa871740640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:41.769 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.757+0000 7fa871740640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:41.769 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.757+0000 7fa871740640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:41.769 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.758+0000 7fa870f3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:41.769 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.758+0000 7fa870f3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:41.769 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.758+0000 7fa870f3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:41.769 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.759+0000 7fa871740640 10 monclient(hunting): handle_auth_done global_id 4234 payload 274 2024-08-09T14:50:41.769 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.759+0000 7fa871740640 1 -- 172.21.15.155:0/3825470129 >> v2:172.21.15.155:3300/0 conn(0x7fa86c0b0190 msgr2=0x7fa86c142db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:41.769 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.759+0000 7fa871740640 1 --2- 172.21.15.155:0/3825470129 >> v2:172.21.15.155:3300/0 conn(0x7fa86c0b0190 0x7fa86c142db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:41.769 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.759+0000 7fa871740640 1 -- 172.21.15.155:0/3825470129 >> v1:172.21.15.155:6789/0 conn(0x7fa86c0acff0 legacy=0x7fa86c13c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:41.770 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.759+0000 7fa871740640 10 monclient: _finish_hunting 0 2024-08-09T14:50:41.770 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.759+0000 7fa871740640 1 monclient: found mon.noname-c 2024-08-09T14:50:41.770 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.759+0000 7fa871740640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:41.770 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.759+0000 7fa871740640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:50:41.770 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.759+0000 7fa871740640 1 -- 172.21.15.155:0/3825470129 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86c0ac7f0 con 0x7fa86c13cf50 2024-08-09T14:50:41.770 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.761+0000 7fa871740640 1 --2- 172.21.15.155:0/3825470129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa86c13cf50 0x7fa86c0ad430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0687d0 tx=0x7fa86402e0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=626d0572bd4ed7cd server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:41.774 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa86b7fe640 1 -- 172.21.15.155:0/3825470129 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86402ea90 con 0x7fa86c13cf50 2024-08-09T14:50:41.775 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa86b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:41.775 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa86b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:41.775 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa86b7fe640 10 monclient: dump: 2024-08-09T14:50:41.775 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:41.775 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:41.775 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:41.775 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:41.775 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:41.775 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:41.775 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:41.775 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:41.775 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:41.776 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:41.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa86b7fe640 10 monclient: _finish_auth 0 2024-08-09T14:50:41.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa86b7fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:41.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa86b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:41.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa86b7fe640 1 -- 172.21.15.155:0/3825470129 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa86402ec30 con 0x7fa86c13cf50 2024-08-09T14:50:41.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa8731a3640 5 monclient: authenticate success, global_id 4234 2024-08-09T14:50:41.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa86b7fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:41.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa86b7fe640 1 -- 172.21.15.155:0/3825470129 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864009600 con 0x7fa86c13cf50 2024-08-09T14:50:41.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa8731a3640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:41.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa8731a3640 4 set_mon_vals no callback set 2024-08-09T14:50:41.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa86b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:41.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa86b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:41.776 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa86b7fe640 10 monclient: dump: 2024-08-09T14:50:41.777 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:41.777 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:41.777 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:41.777 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:41.777 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:41.777 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:41.777 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:41.777 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:41.777 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:41.777 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:41.777 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa8731a3640 10 monclient: shutdown 2024-08-09T14:50:41.777 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa8731a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:41.777 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa8731a3640 1 -- 172.21.15.155:0/3825470129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa86c13cf50 msgr2=0x7fa86c0ad430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:41.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa8731a3640 1 --2- 172.21.15.155:0/3825470129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa86c13cf50 0x7fa86c0ad430 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0687d0 tx=0x7fa86402e0f0 comp rx=0 tx=0).stop 2024-08-09T14:50:41.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa8731a3640 1 -- 172.21.15.155:0/3825470129 shutdown_connections 2024-08-09T14:50:41.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa8731a3640 1 --2- 172.21.15.155:0/3825470129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa86c13cf50 0x7fa86c0ad430 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:41.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa8731a3640 1 --2- 172.21.15.155:0/3825470129 >> v2:172.21.15.155:3300/0 conn(0x7fa86c0b0190 0x7fa86c142db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:41.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa8731a3640 1 -- 172.21.15.155:0/3825470129 >> 172.21.15.155:0/3825470129 conn(0x7fa86c06bf50 msgr2=0x7fa86c06c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:41.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa8731a3640 1 -- 172.21.15.155:0/3825470129 shutdown_connections 2024-08-09T14:50:41.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.765+0000 7fa8731a3640 1 -- 172.21.15.155:0/3825470129 wait complete. 2024-08-09T14:50:41.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.766+0000 7fa8731a3640 10 monclient: build_initial_monmap 2024-08-09T14:50:41.778 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.766+0000 7fa8731a3640 10 monclient: monmap: 2024-08-09T14:50:41.778 INFO:teuthology.misc.health.smithi155.stderr:epoch 0 2024-08-09T14:50:41.778 INFO:teuthology.misc.health.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:41.779 INFO:teuthology.misc.health.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:41.779 INFO:teuthology.misc.health.smithi155.stderr:created 0.000000 2024-08-09T14:50:41.779 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:41.779 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:41.779 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:41.779 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:41.779 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:41.779 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:41.779 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.766+0000 7fa8731a3640 1 librados: starting msgr at 2024-08-09T14:50:41.779 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.766+0000 7fa8731a3640 1 librados: starting objecter 2024-08-09T14:50:41.779 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 1 Processor -- start 2024-08-09T14:50:41.779 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 1 -- start start 2024-08-09T14:50:41.779 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 1 librados: setting wanted keys 2024-08-09T14:50:41.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 1 librados: calling monclient init 2024-08-09T14:50:41.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 10 monclient: init 2024-08-09T14:50:41.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:41.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 10 monclient: _add_conns ranks=[0,2,1] 2024-08-09T14:50:41.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa86c0acff0 0x7fa86c139560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:41.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 10 monclient(hunting): picked mon.noname-a con 0x7fa86c0acff0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:41.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa86c0b0190 0x7fa86c072a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:41.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 10 monclient(hunting): picked mon.noname-c con 0x7fa86c0b0190 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:41.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa86c13cf50 0x7fa86c13d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:41.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 10 monclient(hunting): picked mon.noname-b con 0x7fa86c13cf50 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:41.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:41.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa86c0aa4c0 con 0x7fa86c0acff0 2024-08-09T14:50:41.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:41.780 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa86c073e00 con 0x7fa86c13cf50 2024-08-09T14:50:41.781 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:41.781 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fa86c0aa640 con 0x7fa86c0b0190 2024-08-09T14:50:41.781 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:41.781 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.767+0000 7fa8731a3640 10 monclient(hunting): authenticate will time out at 1248.804831s 2024-08-09T14:50:41.781 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa870f3f640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa86c0acff0 0x7fa86c139560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:41.781 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa871740640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa86c13cf50 0x7fa86c13d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:41.781 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa86bfff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa86c0b0190 0x7fa86c072a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:41.781 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa86bfff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa86c0b0190 0x7fa86c072a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:34200/0 (socket says 172.21.15.155:34200) 2024-08-09T14:50:41.781 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa86bfff640 1 -- 172.21.15.155:0/1970642415 learned_addr learned my addr 172.21.15.155:0/1970642415 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:41.781 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa86bfff640 10 monclient(hunting): get_auth_request con 0x7fa86c0b0190 auth_method 0 2024-08-09T14:50:41.781 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa86bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:41.781 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa86bfff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:41.781 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa86bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:41.782 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa870f3f640 10 monclient(hunting): get_auth_request con 0x7fa86c0acff0 auth_method 0 2024-08-09T14:50:41.782 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa870f3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:41.782 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa870f3f640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:41.782 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa870f3f640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:41.782 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa871740640 10 monclient(hunting): get_auth_request con 0x7fa86c13cf50 auth_method 0 2024-08-09T14:50:41.782 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa871740640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:41.782 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa871740640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:41.782 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.768+0000 7fa871740640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:41.782 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.769+0000 7fa871740640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:41.782 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.769+0000 7fa871740640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:41.782 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.769+0000 7fa871740640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:41.782 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.769+0000 7fa86bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:41.782 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.769+0000 7fa86bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:41.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.769+0000 7fa86bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:41.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.769+0000 7fa870f3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:41.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.769+0000 7fa870f3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:41.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.769+0000 7fa870f3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:41.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.771+0000 7fa871740640 10 monclient(hunting): handle_auth_done global_id 4237 payload 824 2024-08-09T14:50:41.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.771+0000 7fa871740640 1 -- 172.21.15.155:0/1970642415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa86c0b0190 msgr2=0x7fa86c072a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:41.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.771+0000 7fa871740640 1 --2- 172.21.15.155:0/1970642415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa86c0b0190 0x7fa86c072a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:41.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.771+0000 7fa871740640 1 -- 172.21.15.155:0/1970642415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa86c0acff0 msgr2=0x7fa86c139560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:41.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.771+0000 7fa871740640 1 --2- 172.21.15.155:0/1970642415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa86c0acff0 0x7fa86c139560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:41.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.771+0000 7fa871740640 10 monclient: _finish_hunting 0 2024-08-09T14:50:41.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.771+0000 7fa871740640 1 monclient: found mon.noname-b 2024-08-09T14:50:41.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.771+0000 7fa871740640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:41.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.771+0000 7fa871740640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:41.783 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.771+0000 7fa871740640 1 -- 172.21.15.155:0/1970642415 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa86c1d85c0 con 0x7fa86c13cf50 2024-08-09T14:50:41.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.774+0000 7fa871740640 1 --2- 172.21.15.155:0/1970642415 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa86c13cf50 0x7fa86c13d630 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa864034000 tx=0x7fa864004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:41.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.774+0000 7fa8697fa640 10 client.?.objecter ms_handle_connect 0x7fa86c13cf50 2024-08-09T14:50:41.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.774+0000 7fa8697fa640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:41.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.777+0000 7fa8697fa640 1 -- 172.21.15.155:0/1970642415 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86400b720 con 0x7fa86c13cf50 2024-08-09T14:50:41.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.777+0000 7fa8697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:41.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.777+0000 7fa8697fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:41.785 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.777+0000 7fa8697fa640 10 monclient: dump: 2024-08-09T14:50:41.786 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:41.786 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:41.786 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:41.786 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:41.786 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:41.786 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:41.786 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:41.786 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:41.786 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:41.786 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:41.786 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8697fa640 10 monclient: _finish_auth 0 2024-08-09T14:50:41.787 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8697fa640 10 monclient: _check_auth_tickets 2024-08-09T14:50:41.787 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:41.787 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8697fa640 1 -- 172.21.15.155:0/1970642415 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa864009370 con 0x7fa86c13cf50 2024-08-09T14:50:41.787 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8697fa640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:41.787 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8697fa640 1 -- 172.21.15.155:0/1970642415 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864036820 con 0x7fa86c13cf50 2024-08-09T14:50:41.787 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8697fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:41.787 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:41.787 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8697fa640 10 monclient: dump: 2024-08-09T14:50:41.787 INFO:teuthology.misc.health.smithi155.stderr:epoch 1 2024-08-09T14:50:41.787 INFO:teuthology.misc.health.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:41.787 INFO:teuthology.misc.health.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:41.787 INFO:teuthology.misc.health.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:41.788 INFO:teuthology.misc.health.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:41.788 INFO:teuthology.misc.health.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:41.788 INFO:teuthology.misc.health.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:41.788 INFO:teuthology.misc.health.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:41.788 INFO:teuthology.misc.health.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:41.788 INFO:teuthology.misc.health.smithi155.stderr: 2024-08-09T14:50:41.788 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8731a3640 5 monclient: authenticate success, global_id 4237 2024-08-09T14:50:41.788 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8731a3640 10 monclient: _renew_subs 2024-08-09T14:50:41.789 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8731a3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:41.789 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8731a3640 1 -- 172.21.15.155:0/1970642415 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa86c1d8290 con 0x7fa86c13cf50 2024-08-09T14:50:41.789 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa86affd640 4 set_mon_vals no callback set 2024-08-09T14:50:41.789 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8731a3640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:41.789 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8731a3640 10 monclient: _renew_subs 2024-08-09T14:50:41.789 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8731a3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:41.790 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8731a3640 1 -- 172.21.15.155:0/1970642415 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa86c13a1e0 con 0x7fa86c13cf50 2024-08-09T14:50:41.790 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8731a3640 1 librados: init done 2024-08-09T14:50:41.790 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.778+0000 7fa8731a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:41.791 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.779+0000 7fa8731a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:41.791 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.779+0000 7fa8731a3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:41.791 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.779+0000 7fa8731a3640 1 -- 172.21.15.155:0/1970642415 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa86c1430b0 con 0x7fa86c13cf50 2024-08-09T14:50:41.800 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.798+0000 7fa8697fa640 1 -- 172.21.15.155:0/1970642415 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa864047020 con 0x7fa86c13cf50 2024-08-09T14:50:41.800 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.799+0000 7fa8697fa640 1 --2- 172.21.15.155:0/1970642415 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa85006d840 0x7fa85006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:41.800 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.799+0000 7fa870f3f640 1 --2- 172.21.15.155:0/1970642415 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa85006d840 0x7fa85006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:41.801 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.799+0000 7fa870f3f640 10 monclient: get_auth_request con 0x7fa85006d840 auth_method 0 2024-08-09T14:50:41.801 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.799+0000 7fa8697fa640 10 client.4237.objecter ms_handle_connect 0x7fa85006d840 2024-08-09T14:50:41.801 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:41.799+0000 7fa870f3f640 1 --2- 172.21.15.155:0/1970642415 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa85006d840 0x7fa85006fd00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa860009b40 tx=0x7fa860009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:42.025 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.023+0000 7fa8697fa640 1 -- 172.21.15.155:0/1970642415 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 4857+0+0 (secure 0 0 0) 0x7fa864082780 con 0x7fa86c13cf50 2024-08-09T14:50:42.025 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.023+0000 7fa8697fa640 10 client.4237.objecter ms_dispatch 0x7fa86c000b90 osd_map(22..22 src has 1..22) 2024-08-09T14:50:42.025 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.023+0000 7fa8697fa640 3 client.4237.objecter handle_osd_map got epochs [22,22] > 0 2024-08-09T14:50:42.025 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.023+0000 7fa8697fa640 3 client.4237.objecter handle_osd_map decoding full epoch 22 2024-08-09T14:50:42.025 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.023+0000 7fa8697fa640 20 client.4237.objecter dump_active .. 0 homeless 2024-08-09T14:50:42.035 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.034+0000 7fa8697fa640 1 -- 172.21.15.155:0/1970642415 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7fa86c1430b0 con 0x7fa86c13cf50 2024-08-09T14:50:42.035 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.034+0000 7fa8697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:42.035 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.034+0000 7fa8697fa640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:42.223 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.221+0000 7fa8731a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:42.223 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.221+0000 7fa8731a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:42.223 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.221+0000 7fa8731a3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:42.223 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.221+0000 7fa8731a3640 1 -- 172.21.15.155:0/1970642415 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa86c142db0 con 0x7fa86c13cf50 2024-08-09T14:50:42.298 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.296+0000 7fa8697fa640 1 -- 172.21.15.155:0/1970642415 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+12920 (secure 0 0 0) 0x7fa86404e020 con 0x7fa86c13cf50 2024-08-09T14:50:42.298 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.296+0000 7fa8697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:42.298 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.296+0000 7fa8697fa640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:42.298 INFO:teuthology.misc.health.smithi155.stdout: 2024-08-09T14:50:42.299 INFO:teuthology.misc.health.smithi155.stdout:{"epoch":22,"fsid":"f3d21f4d-627f-4083-bf15-ae5f3fe632de","created":"2024-08-09T14:49:41.321598+0000","modified":"2024-08-09T14:50:40.940341+0000","last_up_change":"2024-08-09T14:50:39.237260+0000","last_in_change":"2024-08-09T14:49:46.978521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-09T14:50:30.139923+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6526d474-661b-48e6-bcc7-6288c7477f78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":729861593},{"type":"v1","addr":"172.21.15.155:6822","nonce":729861593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":729861593},{"type":"v1","addr":"172.21.15.155:6828","nonce":729861593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":729861593},{"type":"v1","addr":"172.21.15.155:6833","nonce":729861593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":729861593},{"type":"v1","addr":"172.21.15.155:6831","nonce":729861593}]},"public_addr":"172.21.15.155:6822/729861593","cluster_addr":"172.21.15.155:6828/729861593","heartbeat_back_addr":"172.21.15.155:6833/729861593","heartbeat_front_addr":"172.21.15.155:6831/729861593","state":["exists","up"]},{"osd":1,"uuid":"87673c5a-b124-419e-98da-17a7b7fbdf25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6819","nonce":3129678288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6821","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6823","nonce":3129678288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6827","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6829","nonce":3129678288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6825","nonce":3129678288}]},"public_addr":"172.21.15.155:6819/3129678288","cluster_addr":"172.21.15.155:6823/3129678288","heartbeat_back_addr":"172.21.15.155:6829/3129678288","heartbeat_front_addr":"172.21.15.155:6825/3129678288","state":["exists","up"]},{"osd":2,"uuid":"a14a708a-903f-4de2-8228-79e79ed135e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":961143006},{"type":"v1","addr":"172.21.15.155:6803","nonce":961143006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":961143006},{"type":"v1","addr":"172.21.15.155:6805","nonce":961143006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":961143006},{"type":"v1","addr":"172.21.15.155:6809","nonce":961143006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":961143006},{"type":"v1","addr":"172.21.15.155:6807","nonce":961143006}]},"public_addr":"172.21.15.155:6803/961143006","cluster_addr":"172.21.15.155:6805/961143006","heartbeat_back_addr":"172.21.15.155:6809/961143006","heartbeat_front_addr":"172.21.15.155:6807/961143006","state":["exists","up"]},{"osd":3,"uuid":"8e139d50-e25c-4428-9edf-c696189cb40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6811","nonce":3967029756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6813","nonce":3967029756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6817","nonce":3967029756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6815","nonce":3967029756}]},"public_addr":"172.21.15.155:6811/3967029756","cluster_addr":"172.21.15.155:6813/3967029756","heartbeat_back_addr":"172.21.15.155:6817/3967029756","heartbeat_front_addr":"172.21.15.155:6815/3967029756","state":["exists","up"]},{"osd":4,"uuid":"d0331386-2f50-4bf9-b72e-24c80f9fe9a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6801","nonce":2229949751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6803","nonce":2229949751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6807","nonce":2229949751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6805","nonce":2229949751}]},"public_addr":"172.21.15.194:6801/2229949751","cluster_addr":"172.21.15.194:6803/2229949751","heartbeat_back_addr":"172.21.15.194:6807/2229949751","heartbeat_front_addr":"172.21.15.194:6805/2229949751","state":["exists","up"]},{"osd":5,"uuid":"0f901e55-2ba8-4144-9802-3a109884254c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6817","nonce":1013364343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6819","nonce":1013364343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6825","nonce":1013364343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6821","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6822","nonce":1013364343}]},"public_addr":"172.21.15.194:6817/1013364343","cluster_addr":"172.21.15.194:6819/1013364343","heartbeat_back_addr":"172.21.15.194:6825/1013364343","heartbeat_front_addr":"172.21.15.194:6822/1013364343","state":["exists","up"]},{"osd":6,"uuid":"fd4b8215-814e-4b5a-9877-0ab63818b0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6809","nonce":3077539270}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6811","nonce":3077539270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6815","nonce":3077539270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6813","nonce":3077539270}]},"public_addr":"172.21.15.194:6809/3077539270","cluster_addr":"172.21.15.194:6811/3077539270","heartbeat_back_addr":"172.21.15.194:6815/3077539270","heartbeat_front_addr":"172.21.15.194:6813/3077539270","state":["exists","up"]},{"osd":7,"uuid":"26ed84b6-639d-4ccd-bb1d-c64b593ada7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2670313675},{"type":"v1","addr":"172.21.15.194:6823","nonce":2670313675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2670313675},{"type":"v1","addr":"172.21.15.194:6827","nonce":2670313675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2670313675},{"type":"v1","addr":"172.21.15.194:6831","nonce":2670313675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2670313675},{"type":"v1","addr":"172.21.15.194:6829","nonce":2670313675}]},"public_addr":"172.21.15.194:6823/2670313675","cluster_addr":"172.21.15.194:6827/2670313675","heartbeat_back_addr":"172.21.15.194:6831/2670313675","heartbeat_front_addr":"172.21.15.194:6829/2670313675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:26.771476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:26.207562+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.576525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.163852+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.187804+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:26.108803+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.199884+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.757809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-09T14:50:42.300 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.298+0000 7fa8731a3640 10 librados: watch_flush enter 2024-08-09T14:50:42.301 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.298+0000 7fa8731a3640 10 librados: watch_flush exit 2024-08-09T14:50:42.301 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.298+0000 7fa8731a3640 20 client.4237.objecter shutdown clearing up homeless session... 2024-08-09T14:50:42.301 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.298+0000 7fa8731a3640 10 client.4237.objecter successfully canceled tick 2024-08-09T14:50:42.301 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 1 -- 172.21.15.155:0/1970642415 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa85006d840 msgr2=0x7fa85006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:42.301 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 1 --2- 172.21.15.155:0/1970642415 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa85006d840 0x7fa85006fd00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa860009b40 tx=0x7fa860009190 comp rx=0 tx=0).stop 2024-08-09T14:50:42.301 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 10 monclient: shutdown 2024-08-09T14:50:42.301 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:42.301 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 1 -- 172.21.15.155:0/1970642415 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa86c13cf50 msgr2=0x7fa86c13d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:42.301 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 1 --2- 172.21.15.155:0/1970642415 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa86c13cf50 0x7fa86c13d630 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa864034000 tx=0x7fa864004a40 comp rx=0 tx=0).stop 2024-08-09T14:50:42.301 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 1 -- 172.21.15.155:0/1970642415 shutdown_connections 2024-08-09T14:50:42.301 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 1 --2- 172.21.15.155:0/1970642415 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa85006d840 0x7fa85006fd00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:42.301 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 1 --2- 172.21.15.155:0/1970642415 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa86c13cf50 0x7fa86c13d630 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:42.302 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 1 --2- 172.21.15.155:0/1970642415 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa86c0b0190 0x7fa86c072a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:42.302 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 1 --2- 172.21.15.155:0/1970642415 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa86c0acff0 0x7fa86c139560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:42.302 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 1 -- 172.21.15.155:0/1970642415 >> 172.21.15.155:0/1970642415 conn(0x7fa86c06bf50 msgr2=0x7fa86c0af660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:42.302 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 1 -- 172.21.15.155:0/1970642415 shutdown_connections 2024-08-09T14:50:42.302 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 1 -- 172.21.15.155:0/1970642415 wait complete. 2024-08-09T14:50:42.302 INFO:teuthology.misc.health.smithi155.stderr:2024-08-09T14:50:42.299+0000 7fa8731a3640 1 librados: shutdown 2024-08-09T14:50:42.310 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-08-09T14:50:42.310 INFO:tasks.ceph:Creating RBD pool 2024-08-09T14:50:42.310 DEBUG:teuthology.orchestra.run.smithi155:> sudo ceph --cluster ceph osd pool create rbd 8 2024-08-09T14:50:42.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.407+0000 7f73979b1640 10 monclient: get_monmap_and_config 2024-08-09T14:50:42.409 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.407+0000 7f73979b1640 10 monclient: build_initial_monmap 2024-08-09T14:50:42.409 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.407+0000 7f73979b1640 10 monclient: monmap: 2024-08-09T14:50:42.409 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:42.409 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:42.409 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:42.408563+0000 2024-08-09T14:50:42.409 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:42.408563+0000 2024-08-09T14:50:42.409 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:42.409 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:42.409 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:42.409 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:42.410 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:42.410 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:42.410 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:42.410 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:42.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 1 Processor -- start 2024-08-09T14:50:42.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 1 -- start start 2024-08-09T14:50:42.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 10 monclient: init 2024-08-09T14:50:42.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:42.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-08-09T14:50:42.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f73900b1050 0x7f73900af0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:42.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 10 monclient(hunting): picked mon.noname-b con 0x7f73900b1050 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:42.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f73900b1430 0x7f7390142d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:42.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 10 monclient(hunting): picked mon.noname-a con 0x7f73900b1430 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:42.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7390073c10 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:42.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73900afcf0 con 0x7f7390073c10 2024-08-09T14:50:42.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:42.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f7390073500 con 0x7f73900b1430 2024-08-09T14:50:42.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:42.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f73900af5f0 con 0x7f73900b1050 2024-08-09T14:50:42.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:42.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.408+0000 7f73979b1640 10 monclient(hunting): authenticate will time out at 1249.445947s 2024-08-09T14:50:42.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.409+0000 7f739574d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f73900b1050 0x7f73900af0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:42.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.409+0000 7f7394f4c640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f73900b1430 0x7f7390142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:42.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.409+0000 7f739574d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f73900b1050 0x7f73900af0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:34216/0 (socket says 172.21.15.155:34216) 2024-08-09T14:50:42.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.409+0000 7f739574d640 1 -- 172.21.15.155:0/2791617232 learned_addr learned my addr 172.21.15.155:0/2791617232 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:42.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.409+0000 7f739574d640 10 monclient(hunting): get_auth_request con 0x7f73900b1050 auth_method 0 2024-08-09T14:50:42.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.409+0000 7f739574d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:42.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.409+0000 7f739574d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:42.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.409+0000 7f739574d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:42.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.409+0000 7f7394f4c640 10 monclient(hunting): get_auth_request con 0x7f73900b1430 auth_method 0 2024-08-09T14:50:42.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.410+0000 7f7394f4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:42.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.410+0000 7f7394f4c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:42.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.410+0000 7f7394f4c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:42.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.410+0000 7f739574d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:42.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.410+0000 7f739574d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:42.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.410+0000 7f739574d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:42.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.411+0000 7f7394f4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:42.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.411+0000 7f7394f4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:42.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.411+0000 7f7394f4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:42.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.411+0000 7f739574d640 10 monclient(hunting): handle_auth_done global_id 4238 payload 274 2024-08-09T14:50:42.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.412+0000 7f739574d640 1 -- 172.21.15.155:0/2791617232 >> v2:172.21.15.155:3300/0 conn(0x7f73900b1430 msgr2=0x7f7390142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:42.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.412+0000 7f739574d640 1 --2- 172.21.15.155:0/2791617232 >> v2:172.21.15.155:3300/0 conn(0x7f73900b1430 0x7f7390142d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:42.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.412+0000 7f739574d640 1 -- 172.21.15.155:0/2791617232 >> v1:172.21.15.155:6789/0 conn(0x7f7390073c10 legacy=0x7f7390072df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:42.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.412+0000 7f739574d640 10 monclient: _finish_hunting 0 2024-08-09T14:50:42.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.412+0000 7f739574d640 1 monclient: found mon.noname-b 2024-08-09T14:50:42.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.412+0000 7f739574d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:42.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.412+0000 7f739574d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:42.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.412+0000 7f739574d640 1 -- 172.21.15.155:0/2791617232 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739006fef0 con 0x7f73900b1050 2024-08-09T14:50:42.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.414+0000 7f739574d640 1 --2- 172.21.15.155:0/2791617232 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f73900b1050 0x7f73900af0b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f738c009870 tx=0x7f738c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=831572c95df25b1a server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:42.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.416+0000 7f7387fff640 1 -- 172.21.15.155:0/2791617232 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c035020 con 0x7f73900b1050 2024-08-09T14:50:42.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.416+0000 7f7387fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:42.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.416+0000 7f7387fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:42.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.416+0000 7f7387fff640 10 monclient: dump: 2024-08-09T14:50:42.417 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:42.417 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:42.417 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:42.417 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:42.417 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:42.417 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:42.418 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:42.418 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:42.418 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:42.418 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:42.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.416+0000 7f7387fff640 10 monclient: _finish_auth 0 2024-08-09T14:50:42.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.416+0000 7f7387fff640 10 monclient: _check_auth_tickets 2024-08-09T14:50:42.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.416+0000 7f7387fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:42.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.416+0000 7f73979b1640 5 monclient: authenticate success, global_id 4238 2024-08-09T14:50:42.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.416+0000 7f73979b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:42.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.418+0000 7f7387fff640 1 -- 172.21.15.155:0/2791617232 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f738c031640 con 0x7f73900b1050 2024-08-09T14:50:42.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.418+0000 7f7387fff640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:42.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.418+0000 7f7387fff640 1 -- 172.21.15.155:0/2791617232 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c02d040 con 0x7f73900b1050 2024-08-09T14:50:42.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.418+0000 7f7387fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:42.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.418+0000 7f7387fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:42.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.418+0000 7f7387fff640 10 monclient: dump: 2024-08-09T14:50:42.419 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:42.420 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:42.420 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:42.420 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:42.420 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:42.420 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:42.420 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:42.420 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:42.420 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:42.420 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:42.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.418+0000 7f73979b1640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:42.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.418+0000 7f73979b1640 4 set_mon_vals no callback set 2024-08-09T14:50:42.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.418+0000 7f73979b1640 10 monclient: shutdown 2024-08-09T14:50:42.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.418+0000 7f73979b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:42.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.418+0000 7f73979b1640 1 -- 172.21.15.155:0/2791617232 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f73900b1050 msgr2=0x7f73900af0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:42.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.418+0000 7f73979b1640 1 --2- 172.21.15.155:0/2791617232 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f73900b1050 0x7f73900af0b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f738c009870 tx=0x7f738c02b8f0 comp rx=0 tx=0).stop 2024-08-09T14:50:42.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.419+0000 7f73979b1640 1 -- 172.21.15.155:0/2791617232 shutdown_connections 2024-08-09T14:50:42.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.419+0000 7f73979b1640 1 --2- 172.21.15.155:0/2791617232 >> v2:172.21.15.155:3300/0 conn(0x7f73900b1430 0x7f7390142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:42.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.419+0000 7f73979b1640 1 --2- 172.21.15.155:0/2791617232 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f73900b1050 0x7f73900af0b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:42.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.419+0000 7f73979b1640 1 -- 172.21.15.155:0/2791617232 >> 172.21.15.155:0/2791617232 conn(0x7f739006cc00 msgr2=0x7f739006d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:42.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.419+0000 7f73979b1640 1 -- 172.21.15.155:0/2791617232 shutdown_connections 2024-08-09T14:50:42.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.419+0000 7f73979b1640 1 -- 172.21.15.155:0/2791617232 wait complete. 2024-08-09T14:50:42.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.419+0000 7f73979b1640 10 monclient: build_initial_monmap 2024-08-09T14:50:42.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.419+0000 7f73979b1640 10 monclient: monmap: 2024-08-09T14:50:42.424 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:42.424 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:42.424 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:42.424 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:42.425 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:42.425 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:42.425 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:42.425 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:42.425 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:42.425 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:42.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 1 librados: starting msgr at 2024-08-09T14:50:42.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 1 librados: starting objecter 2024-08-09T14:50:42.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 1 Processor -- start 2024-08-09T14:50:42.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 1 -- start start 2024-08-09T14:50:42.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 1 librados: setting wanted keys 2024-08-09T14:50:42.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 1 librados: calling monclient init 2024-08-09T14:50:42.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 10 monclient: init 2024-08-09T14:50:42.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:42.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 10 monclient: _add_conns ranks=[2,1,0] 2024-08-09T14:50:42.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7390073c10 0x7f739014ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:42.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 10 monclient(hunting): picked mon.noname-c con 0x7f7390073c10 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:42.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73900b1050 0x7f7390149df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:42.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 10 monclient(hunting): picked mon.noname-b con 0x7f73900b1050 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:42.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73901473c0 0x7f73901db610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:42.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 10 monclient(hunting): picked mon.noname-a con 0x7f73901473c0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:42.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:42.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f73900af950 con 0x7f73901473c0 2024-08-09T14:50:42.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:42.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f73900af7d0 con 0x7f73900b1050 2024-08-09T14:50:42.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:42.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f73900afad0 con 0x7f7390073c10 2024-08-09T14:50:42.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:42.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.420+0000 7f73979b1640 10 monclient(hunting): authenticate will time out at 1249.457939s 2024-08-09T14:50:42.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f739574d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7390073c10 0x7f739014ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:42.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f7395f4e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73901473c0 0x7f73901db610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:42.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f7394f4c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73900b1050 0x7f7390149df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:42.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f739574d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7390073c10 0x7f739014ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:34228/0 (socket says 172.21.15.155:34228) 2024-08-09T14:50:42.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f739574d640 1 -- 172.21.15.155:0/4216160998 learned_addr learned my addr 172.21.15.155:0/4216160998 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:42.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f739574d640 10 monclient(hunting): get_auth_request con 0x7f7390073c10 auth_method 0 2024-08-09T14:50:42.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f739574d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:42.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f739574d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:42.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f739574d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:42.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f7395f4e640 10 monclient(hunting): get_auth_request con 0x7f73901473c0 auth_method 0 2024-08-09T14:50:42.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f7395f4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:42.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f7395f4e640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:42.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f7395f4e640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:42.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f7394f4c640 10 monclient(hunting): get_auth_request con 0x7f73900b1050 auth_method 0 2024-08-09T14:50:42.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f7394f4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:42.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f7394f4c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:42.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.421+0000 7f7394f4c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:42.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.422+0000 7f739574d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:42.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.422+0000 7f739574d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:42.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.422+0000 7f739574d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:42.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.422+0000 7f7394f4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:42.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.422+0000 7f7394f4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:42.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.422+0000 7f7394f4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:42.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.422+0000 7f7395f4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:42.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.422+0000 7f7395f4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:42.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.422+0000 7f7395f4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:42.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.424+0000 7f739574d640 10 monclient(hunting): handle_auth_done global_id 4241 payload 824 2024-08-09T14:50:42.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.425+0000 7f739574d640 1 -- 172.21.15.155:0/4216160998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73900b1050 msgr2=0x7f7390149df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:42.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.425+0000 7f739574d640 1 --2- 172.21.15.155:0/4216160998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73900b1050 0x7f7390149df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:42.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.425+0000 7f739574d640 1 -- 172.21.15.155:0/4216160998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73901473c0 msgr2=0x7f73901db610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:42.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.425+0000 7f739574d640 1 --2- 172.21.15.155:0/4216160998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73901473c0 0x7f73901db610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:42.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.425+0000 7f739574d640 10 monclient: _finish_hunting 0 2024-08-09T14:50:42.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.425+0000 7f739574d640 1 monclient: found mon.noname-c 2024-08-09T14:50:42.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.425+0000 7f739574d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:42.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.425+0000 7f739574d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:50:42.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.425+0000 7f739574d640 1 -- 172.21.15.155:0/4216160998 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739014a730 con 0x7f7390073c10 2024-08-09T14:50:42.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.427+0000 7f739574d640 1 --2- 172.21.15.155:0/4216160998 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7390073c10 0x7f739014ac10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f738c000c00 tx=0x7f738c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:42.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.427+0000 7f7385ffb640 10 client.?.objecter ms_handle_connect 0x7f7390073c10 2024-08-09T14:50:42.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.427+0000 7f7385ffb640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:42.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.430+0000 7f7385ffb640 1 -- 172.21.15.155:0/4216160998 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c040070 con 0x7f7390073c10 2024-08-09T14:50:42.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.430+0000 7f7385ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:42.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.430+0000 7f7385ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:42.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.430+0000 7f7385ffb640 10 monclient: dump: 2024-08-09T14:50:42.433 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:42.433 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:42.433 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:42.433 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:42.433 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:42.433 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:42.433 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:42.433 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:42.433 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:42.433 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:42.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.430+0000 7f7385ffb640 10 monclient: _finish_auth 0 2024-08-09T14:50:42.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.430+0000 7f7385ffb640 10 monclient: _check_auth_tickets 2024-08-09T14:50:42.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.430+0000 7f7385ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:42.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.430+0000 7f7385ffb640 1 -- 172.21.15.155:0/4216160998 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f738c0399f0 con 0x7f7390073c10 2024-08-09T14:50:42.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f73979b1640 5 monclient: authenticate success, global_id 4241 2024-08-09T14:50:42.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f73979b1640 10 monclient: _renew_subs 2024-08-09T14:50:42.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f73979b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:42.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f73979b1640 1 -- 172.21.15.155:0/4216160998 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f739014bc20 con 0x7f7390073c10 2024-08-09T14:50:42.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f7385ffb640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:42.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f7385ffb640 1 -- 172.21.15.155:0/4216160998 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c035040 con 0x7f7390073c10 2024-08-09T14:50:42.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f7385ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:42.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f7385ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:42.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f7385ffb640 10 monclient: dump: 2024-08-09T14:50:42.435 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:42.435 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:42.435 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:42.435 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:42.435 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:42.435 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:42.435 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:42.435 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:42.435 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:42.435 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:42.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f73877fe640 4 set_mon_vals no callback set 2024-08-09T14:50:42.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f73979b1640 10 client.4241.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:42.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f73979b1640 10 monclient: _renew_subs 2024-08-09T14:50:42.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f73979b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:42.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f73979b1640 1 -- 172.21.15.155:0/4216160998 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f739014a330 con 0x7f7390073c10 2024-08-09T14:50:42.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.431+0000 7f73979b1640 1 librados: init done 2024-08-09T14:50:42.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.432+0000 7f73979b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:42.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.432+0000 7f73979b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:42.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.432+0000 7f73979b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:42.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.432+0000 7f73979b1640 1 -- 172.21.15.155:0/4216160998 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73900b3050 con 0x7f7390073c10 2024-08-09T14:50:42.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.452+0000 7f7385ffb640 1 -- 172.21.15.155:0/4216160998 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f738c04b060 con 0x7f7390073c10 2024-08-09T14:50:42.454 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.453+0000 7f7385ffb640 1 --2- 172.21.15.155:0/4216160998 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f737406d810 0x7f737406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:42.454 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.453+0000 7f7394f4c640 1 --2- 172.21.15.155:0/4216160998 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f737406d810 0x7f737406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:42.454 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.453+0000 7f7394f4c640 10 monclient: get_auth_request con 0x7f737406d810 auth_method 0 2024-08-09T14:50:42.454 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.453+0000 7f7394f4c640 1 --2- 172.21.15.155:0/4216160998 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f737406d810 0x7f737406fcd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f739014c890 tx=0x7f7380009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:42.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.453+0000 7f7385ffb640 10 client.4241.objecter ms_handle_connect 0x7f737406d810 2024-08-09T14:50:42.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.455+0000 7f7385ffb640 1 -- 172.21.15.155:0/4216160998 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 4857+0+0 (secure 0 0 0) 0x7f738c0815e0 con 0x7f7390073c10 2024-08-09T14:50:42.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.455+0000 7f7385ffb640 10 client.4241.objecter ms_dispatch 0x7f7390000b90 osd_map(22..22 src has 1..22) 2024-08-09T14:50:42.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.455+0000 7f7385ffb640 3 client.4241.objecter handle_osd_map got epochs [22,22] > 0 2024-08-09T14:50:42.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.455+0000 7f7385ffb640 3 client.4241.objecter handle_osd_map decoding full epoch 22 2024-08-09T14:50:42.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.456+0000 7f7385ffb640 20 client.4241.objecter dump_active .. 0 homeless 2024-08-09T14:50:42.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.702+0000 7f7385ffb640 1 -- 172.21.15.155:0/4216160998 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f73900b3050 con 0x7f7390073c10 2024-08-09T14:50:42.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.702+0000 7f7385ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:42.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.702+0000 7f7385ffb640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:42.884 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.883+0000 7f73979b1640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-08-09T14:50:42.884 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.883+0000 7f73979b1640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-08-09T14:50:42.885 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.883+0000 7f73979b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:42.885 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:42.883+0000 7f73979b1640 1 -- 172.21.15.155:0/4216160998 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f73900af0b0 con 0x7f7390073c10 2024-08-09T14:50:43.319 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.318+0000 7f7385ffb640 1 -- 172.21.15.155:0/4216160998 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v23) ==== 109+0+0 (secure 0 0 0) 0x7f738c07adf0 con 0x7f7390073c10 2024-08-09T14:50:43.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.318+0000 7f7385ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-08-09T14:50:43.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.318+0000 7f7385ffb640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-08-09T14:50:43.320 INFO:teuthology.orchestra.run.smithi155.stderr:pool 'rbd' created 2024-08-09T14:50:43.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 10 librados: watch_flush enter 2024-08-09T14:50:43.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 10 librados: watch_flush exit 2024-08-09T14:50:43.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 20 client.4241.objecter shutdown clearing up homeless session... 2024-08-09T14:50:43.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 10 client.4241.objecter successfully canceled tick 2024-08-09T14:50:43.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 1 -- 172.21.15.155:0/4216160998 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f737406d810 msgr2=0x7f737406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:43.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 1 --2- 172.21.15.155:0/4216160998 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f737406d810 0x7f737406fcd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f739014c890 tx=0x7f7380009190 comp rx=0 tx=0).stop 2024-08-09T14:50:43.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 10 monclient: shutdown 2024-08-09T14:50:43.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:43.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 1 -- 172.21.15.155:0/4216160998 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7390073c10 msgr2=0x7f739014ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:43.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 1 --2- 172.21.15.155:0/4216160998 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7390073c10 0x7f739014ac10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f738c000c00 tx=0x7f738c0046c0 comp rx=0 tx=0).stop 2024-08-09T14:50:43.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 1 -- 172.21.15.155:0/4216160998 shutdown_connections 2024-08-09T14:50:43.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 1 --2- 172.21.15.155:0/4216160998 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f737406d810 0x7f737406fcd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:43.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 1 --2- 172.21.15.155:0/4216160998 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f73901473c0 0x7f73901db610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:43.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 1 --2- 172.21.15.155:0/4216160998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73900b1050 0x7f7390149df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:43.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 1 --2- 172.21.15.155:0/4216160998 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f7390073c10 0x7f739014ac10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:43.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 1 -- 172.21.15.155:0/4216160998 >> 172.21.15.155:0/4216160998 conn(0x7f739006cc00 msgr2=0x7f739006b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:43.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 1 -- 172.21.15.155:0/4216160998 shutdown_connections 2024-08-09T14:50:43.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 1 -- 172.21.15.155:0/4216160998 wait complete. 2024-08-09T14:50:43.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.321+0000 7f73979b1640 1 librados: shutdown 2024-08-09T14:50:43.334 DEBUG:teuthology.orchestra.run.smithi155:> rbd --cluster ceph pool init rbd 2024-08-09T14:50:43.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.682+0000 7f73e1ea0d80 10 monclient: get_monmap_and_config 2024-08-09T14:50:43.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.682+0000 7f73e1ea0d80 10 monclient: build_initial_monmap 2024-08-09T14:50:43.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.682+0000 7f73e1ea0d80 10 monclient: monmap: 2024-08-09T14:50:43.684 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:43.684 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:43.684 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:43.683819+0000 2024-08-09T14:50:43.684 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:43.683819+0000 2024-08-09T14:50:43.684 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:43.684 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:43.684 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:43.685 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:43.685 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:43.685 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:43.685 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:43.685 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:43.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.683+0000 7f73e1ea0d80 1 Processor -- start 2024-08-09T14:50:43.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.683+0000 7f73e1ea0d80 1 -- start start 2024-08-09T14:50:43.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.683+0000 7f73e1ea0d80 10 monclient: init 2024-08-09T14:50:43.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73e1ea0d80 10 monclient: _reopen_session rank -1 2024-08-09T14:50:43.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73e1ea0d80 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-08-09T14:50:43.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73e1ea0d80 10 monclient(hunting): picked mon.noname-c-legacy con 0x5576d5f87790 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:43.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73e1ea0d80 1 --2- >> v2:172.21.15.155:3300/0 conn(0x5576d5faa4b0 0x5576d604a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:43.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73e1ea0d80 10 monclient(hunting): picked mon.noname-a con 0x5576d5faa4b0 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:43.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73e1ea0d80 10 monclient(hunting): picked mon.noname-a-legacy con 0x5576d604a9c0 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:43.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73e1ea0d80 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5576d5e06ca0 con 0x5576d604a9c0 2024-08-09T14:50:43.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73e1ea0d80 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x5576d5dff4f0 con 0x5576d5f87790 2024-08-09T14:50:43.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73e1ea0d80 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:43.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73e1ea0d80 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x5576d5e104d0 con 0x5576d5faa4b0 2024-08-09T14:50:43.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73e1ea0d80 10 monclient(hunting): _renew_subs 2024-08-09T14:50:43.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73e1ea0d80 10 monclient(hunting): authenticate will time out at 1250.721185s 2024-08-09T14:50:43.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73dbfff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x5576d5faa4b0 0x5576d604a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:43.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73e099a640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x5576d5f87790 0x5576d5fb2da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.155:50184/0 (socket says 172.21.15.155:50184) 2024-08-09T14:50:43.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.684+0000 7f73e099a640 1 -- 172.21.15.155:0/2010333587 learned_addr learned my addr 172.21.15.155:0/2010333587 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:43.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.685+0000 7f73dbfff640 10 monclient(hunting): get_auth_request con 0x5576d5faa4b0 auth_method 0 2024-08-09T14:50:43.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.685+0000 7f73dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:43.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.685+0000 7f73dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:43.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.685+0000 7f73dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:43.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.686+0000 7f73dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:43.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.686+0000 7f73dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:43.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.686+0000 7f73dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:43.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.687+0000 7f73db7fe640 1 -- 172.21.15.155:0/2010333587 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3259370316 0 0) 0x5576d5dff4f0 con 0x5576d5f87790 2024-08-09T14:50:43.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.687+0000 7f73db7fe640 10 monclient(hunting): _negotiate 2024-08-09T14:50:43.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.687+0000 7f73db7fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:43.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.687+0000 7f73db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:43.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.687+0000 7f73db7fe640 10 monclient(hunting): my global_id is 4243 2024-08-09T14:50:43.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.687+0000 7f73db7fe640 1 -- 172.21.15.155:0/2010333587 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73c4001e10 con 0x5576d5f87790 2024-08-09T14:50:43.691 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.690+0000 7f73dbfff640 10 monclient(hunting): handle_auth_done global_id 4263 payload 274 2024-08-09T14:50:43.691 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.690+0000 7f73dbfff640 1 -- 172.21.15.155:0/2010333587 >> v1:172.21.15.194:6789/0 conn(0x5576d5f87790 legacy=0x5576d5fb2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:43.691 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.690+0000 7f73dbfff640 1 -- 172.21.15.155:0/2010333587 >> v1:172.21.15.155:6789/0 conn(0x5576d604a9c0 legacy=0x5576d60514c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:43.691 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.690+0000 7f73dbfff640 10 monclient: _finish_hunting 0 2024-08-09T14:50:43.691 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.690+0000 7f73dbfff640 1 monclient: found mon.noname-a 2024-08-09T14:50:43.692 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.690+0000 7f73dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:43.692 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.690+0000 7f73dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:43.692 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.690+0000 7f73dbfff640 1 -- 172.21.15.155:0/2010333587 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576d5df5ce0 con 0x5576d5faa4b0 2024-08-09T14:50:43.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.694+0000 7f73dbfff640 1 --2- 172.21.15.155:0/2010333587 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576d5faa4b0 0x5576d604a480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f73d000a4f0 tx=0x7f73d001ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f38190fdb0b225c server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:43.698 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.696+0000 7f73db7fe640 1 -- 172.21.15.155:0/2010333587 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d0057070 con 0x5576d5faa4b0 2024-08-09T14:50:43.698 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.696+0000 7f73db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:43.698 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.696+0000 7f73db7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:43.698 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.696+0000 7f73db7fe640 10 monclient: dump: 2024-08-09T14:50:43.698 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:43.698 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:43.698 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:43.699 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:43.699 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:43.699 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:43.699 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:43.699 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:43.699 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:43.699 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:43.699 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.696+0000 7f73db7fe640 10 monclient: _finish_auth 0 2024-08-09T14:50:43.699 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.696+0000 7f73db7fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:43.699 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.696+0000 7f73db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:43.699 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.696+0000 7f73e1ea0d80 5 monclient: authenticate success, global_id 4263 2024-08-09T14:50:43.699 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.696+0000 7f73e1ea0d80 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:43.701 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.698+0000 7f73db7fe640 1 -- 172.21.15.155:0/2010333587 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d004ad80 con 0x5576d5faa4b0 2024-08-09T14:50:43.701 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.698+0000 7f73db7fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:43.701 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.698+0000 7f73e1ea0d80 10 monclient: get_monmap_and_config success 2024-08-09T14:50:43.701 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.698+0000 7f73e1ea0d80 4 set_mon_vals no callback set 2024-08-09T14:50:43.701 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.699+0000 7f73db7fe640 1 -- 172.21.15.155:0/2010333587 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d0052640 con 0x5576d5faa4b0 2024-08-09T14:50:43.702 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.699+0000 7f73db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:43.702 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.699+0000 7f73db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:43.702 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.699+0000 7f73db7fe640 10 monclient: dump: 2024-08-09T14:50:43.702 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:43.702 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:43.702 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:43.702 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:43.702 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:43.702 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:43.702 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:43.702 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:43.702 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:43.703 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:43.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.699+0000 7f73e1ea0d80 10 monclient: shutdown 2024-08-09T14:50:43.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.699+0000 7f73e1ea0d80 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:43.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.699+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/2010333587 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576d5faa4b0 msgr2=0x5576d604a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:43.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.699+0000 7f73e1ea0d80 1 --2- 172.21.15.155:0/2010333587 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576d5faa4b0 0x5576d604a480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f73d000a4f0 tx=0x7f73d001ce30 comp rx=0 tx=0).stop 2024-08-09T14:50:43.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.699+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/2010333587 shutdown_connections 2024-08-09T14:50:43.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.699+0000 7f73e1ea0d80 1 --2- 172.21.15.155:0/2010333587 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576d5faa4b0 0x5576d604a480 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:43.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.699+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/2010333587 >> 172.21.15.155:0/2010333587 conn(0x5576d5f88d20 msgr2=0x5576d5fb60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:43.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.700+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/2010333587 shutdown_connections 2024-08-09T14:50:43.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.700+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/2010333587 wait complete. 2024-08-09T14:50:43.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.701+0000 7f73e1ea0d80 10 monclient: build_initial_monmap 2024-08-09T14:50:43.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.701+0000 7f73e1ea0d80 10 monclient: monmap: 2024-08-09T14:50:43.705 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:43.706 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:43.706 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:43.706 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:43.706 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:43.706 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:43.706 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:43.706 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:43.706 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:43.706 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:43.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.701+0000 7f73e1ea0d80 1 librados: starting msgr at 2024-08-09T14:50:43.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.701+0000 7f73e1ea0d80 1 librados: starting objecter 2024-08-09T14:50:43.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.701+0000 7f73e1ea0d80 1 Processor -- start 2024-08-09T14:50:43.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.701+0000 7f73e1ea0d80 1 -- start start 2024-08-09T14:50:43.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.701+0000 7f73e1ea0d80 1 librados: setting wanted keys 2024-08-09T14:50:43.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.701+0000 7f73e1ea0d80 1 librados: calling monclient init 2024-08-09T14:50:43.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.701+0000 7f73e1ea0d80 10 monclient: init 2024-08-09T14:50:43.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 10 monclient: _reopen_session rank -1 2024-08-09T14:50:43.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 10 monclient: _add_conns ranks=[2,0,1] 2024-08-09T14:50:43.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x5576d5f87790 0x5576d6040af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:43.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 10 monclient(hunting): picked mon.noname-c con 0x5576d5f87790 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:43.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576d5faa4b0 0x5576d60417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:43.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 10 monclient(hunting): picked mon.noname-a con 0x5576d5faa4b0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:43.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d604a9c0 0x5576d6041d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:43.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 10 monclient(hunting): picked mon.noname-b con 0x5576d604a9c0 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:43.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:43.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x5576d5e70eb0 con 0x5576d5faa4b0 2024-08-09T14:50:43.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:43.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5576d5e74ef0 con 0x5576d604a9c0 2024-08-09T14:50:43.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:43.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x5576d5e1bc40 con 0x5576d5f87790 2024-08-09T14:50:43.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 10 monclient(hunting): _renew_subs 2024-08-09T14:50:43.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.702+0000 7f73e1ea0d80 10 monclient(hunting): authenticate will time out at 1250.739774s 2024-08-09T14:50:43.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.703+0000 7f73dbfff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576d5faa4b0 0x5576d60417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:43.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.703+0000 7f73e099a640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x5576d5f87790 0x5576d6040af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:43.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.703+0000 7f73dbfff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576d5faa4b0 0x5576d60417d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:41232/0 (socket says 172.21.15.155:41232) 2024-08-09T14:50:43.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.703+0000 7f73dbfff640 1 -- 172.21.15.155:0/3082486450 learned_addr learned my addr 172.21.15.155:0/3082486450 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:43.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.703+0000 7f73dbfff640 10 monclient(hunting): get_auth_request con 0x5576d5faa4b0 auth_method 0 2024-08-09T14:50:43.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.703+0000 7f73dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:43.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.703+0000 7f73dbfff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:43.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.703+0000 7f73dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:43.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.703+0000 7f73e099a640 10 monclient(hunting): get_auth_request con 0x5576d5f87790 auth_method 0 2024-08-09T14:50:43.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.703+0000 7f73e099a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:43.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.703+0000 7f73e099a640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:43.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.703+0000 7f73e099a640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:43.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.704+0000 7f73dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:43.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.704+0000 7f73dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:43.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.704+0000 7f73dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:43.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.705+0000 7f73e099a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:43.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.705+0000 7f73e099a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:43.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.705+0000 7f73e099a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:43.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.707+0000 7f73dbfff640 10 monclient(hunting): handle_auth_done global_id 4269 payload 824 2024-08-09T14:50:43.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.707+0000 7f73dbfff640 1 -- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x5576d5f87790 msgr2=0x5576d6040af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:43.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.707+0000 7f73dbfff640 1 --2- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x5576d5f87790 0x5576d6040af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:43.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.707+0000 7f73dbfff640 1 -- 172.21.15.155:0/3082486450 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d604a9c0 msgr2=0x5576d6041d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:43.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.707+0000 7f73dbfff640 1 --2- 172.21.15.155:0/3082486450 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576d604a9c0 0x5576d6041d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:43.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.707+0000 7f73dbfff640 10 monclient: _finish_hunting 0 2024-08-09T14:50:43.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.707+0000 7f73dbfff640 1 monclient: found mon.noname-a 2024-08-09T14:50:43.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.707+0000 7f73dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:43.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.707+0000 7f73dbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:43.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.707+0000 7f73dbfff640 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576d6043910 con 0x5576d5faa4b0 2024-08-09T14:50:43.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.712+0000 7f73dbfff640 1 --2- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576d5faa4b0 0x5576d60417d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f73d000a4f0 tx=0x7f73d0002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:43.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.712+0000 7f73d8ff9640 10 client.?.objecter ms_handle_connect 0x5576d5faa4b0 2024-08-09T14:50:43.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.712+0000 7f73d8ff9640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:43.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d0057050 con 0x5576d5faa4b0 2024-08-09T14:50:43.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:43.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73d8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:43.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73d8ff9640 10 monclient: dump: 2024-08-09T14:50:43.724 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:43.724 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:43.724 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:43.724 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:43.725 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:43.725 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:43.725 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:43.725 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:43.725 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:43.725 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:43.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73d8ff9640 10 monclient: _finish_auth 0 2024-08-09T14:50:43.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73d8ff9640 10 monclient: _check_auth_tickets 2024-08-09T14:50:43.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:43.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73d0063070 con 0x5576d5faa4b0 2024-08-09T14:50:43.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73e1ea0d80 5 monclient: authenticate success, global_id 4269 2024-08-09T14:50:43.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73e1ea0d80 10 monclient: _renew_subs 2024-08-09T14:50:43.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73e1ea0d80 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:43.726 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x5576d60434e0 con 0x5576d5faa4b0 2024-08-09T14:50:43.726 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73d8ff9640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:43.726 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d0052af0 con 0x5576d5faa4b0 2024-08-09T14:50:43.726 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:43.726 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73d8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:43.726 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73d8ff9640 10 monclient: dump: 2024-08-09T14:50:43.726 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:43.726 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:43.726 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:43.726 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:43.726 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:43.726 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:43.727 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:43.727 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:43.727 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:43.727 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:43.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73daffd640 4 set_mon_vals no callback set 2024-08-09T14:50:43.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73e1ea0d80 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:43.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73e1ea0d80 10 monclient: _renew_subs 2024-08-09T14:50:43.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73e1ea0d80 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:43.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576d605b720 con 0x5576d5faa4b0 2024-08-09T14:50:43.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73e1ea0d80 1 librados: init done 2024-08-09T14:50:43.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.722+0000 7f73e1ea0d80 10 librados: wait_for_osdmap waiting 2024-08-09T14:50:43.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.748+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 <== mon.0 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f73d0052c90 con 0x5576d5faa4b0 2024-08-09T14:50:43.750 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.749+0000 7f73d8ff9640 1 --2- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f73c006b940 0x7f73c006de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:43.750 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.749+0000 7f73e099a640 1 --2- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f73c006b940 0x7f73c006de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:43.750 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.749+0000 7f73e099a640 10 monclient: get_auth_request con 0x7f73c006b940 auth_method 0 2024-08-09T14:50:43.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.749+0000 7f73d8ff9640 10 client.4269.objecter ms_handle_connect 0x7f73c006b940 2024-08-09T14:50:43.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.749+0000 7f73e099a640 1 --2- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f73c006b940 0x7f73c006de00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f73cc006fd0 tx=0x7f73cc0069a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:43.754 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.752+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 <== mon.0 v2:172.21.15.155:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5199+0+0 (secure 0 0 0) 0x7f73d0097bd0 con 0x5576d5faa4b0 2024-08-09T14:50:43.754 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.752+0000 7f73d8ff9640 10 client.4269.objecter ms_dispatch 0x5576d5e8bc50 osd_map(23..23 src has 1..23) 2024-08-09T14:50:43.754 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.752+0000 7f73d8ff9640 3 client.4269.objecter handle_osd_map got epochs [23,23] > 0 2024-08-09T14:50:43.754 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.752+0000 7f73d8ff9640 3 client.4269.objecter handle_osd_map decoding full epoch 23 2024-08-09T14:50:43.754 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.752+0000 7f73d8ff9640 20 client.4269.objecter dump_active .. 0 homeless 2024-08-09T14:50:43.754 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.752+0000 7f73e1ea0d80 10 librados: wait_for_osdmap done waiting 2024-08-09T14:50:43.754 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.752+0000 7f73e1ea0d80 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-08-09T14:50:43.755 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.752+0000 7f73e1ea0d80 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-08-09T14:50:43.755 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.752+0000 7f73e1ea0d80 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:43.755 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:43.752+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x5576d603e8f0 con 0x5576d5faa4b0 2024-08-09T14:50:44.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.493+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 <== mon.0 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v24) ==== 141+0+0 (secure 0 0 0) 0x7f73d0014ee0 con 0x5576d5faa4b0 2024-08-09T14:50:44.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.493+0000 7f73d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-08-09T14:50:44.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.493+0000 7f73d8ff9640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-08-09T14:50:44.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.493+0000 7f73e1ea0d80 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:44.496 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.493+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f73d004ad80 con 0x5576d5faa4b0 2024-08-09T14:50:44.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.517+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 <== mon.0 v2:172.21.15.155:3300/0 7 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f73d009ae60 con 0x5576d5faa4b0 2024-08-09T14:50:44.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.517+0000 7f73d8ff9640 10 monclient: handle_get_version_reply finishing 1 version 24 2024-08-09T14:50:44.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.517+0000 7f73daffd640 10 client.4269.objecter _get_latest_version latest 24, waiting 2024-08-09T14:50:44.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.517+0000 7f73daffd640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:44.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.517+0000 7f73daffd640 10 monclient: _renew_subs 2024-08-09T14:50:44.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.517+0000 7f73daffd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:44.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.517+0000 7f73daffd640 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=24}) -- 0x7f73b8003c90 con 0x5576d5faa4b0 2024-08-09T14:50:44.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 <== mon.0 v2:172.21.15.155:3300/0 8 ==== osd_map(24..24 src has 1..24) ==== 658+0+0 (secure 0 0 0) 0x7f73d0097e30 con 0x5576d5faa4b0 2024-08-09T14:50:44.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73d8ff9640 10 client.4269.objecter ms_dispatch 0x5576d5e8bc50 osd_map(24..24 src has 1..24) 2024-08-09T14:50:44.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73d8ff9640 3 client.4269.objecter handle_osd_map got epochs [24,24] > 23 2024-08-09T14:50:44.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73d8ff9640 3 client.4269.objecter handle_osd_map decoding incremental epoch 24 2024-08-09T14:50:44.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73d8ff9640 20 client.4269.objecter dump_active .. 0 homeless 2024-08-09T14:50:44.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73e1ea0d80 10 librados: create oid=rbd_trash nspace= 2024-08-09T14:50:44.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73e1ea0d80 10 client.4269.objecter _op_submit op 0x5576d603eff0 2024-08-09T14:50:44.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73e1ea0d80 20 client.4269.objecter _calc_target epoch 24 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:50:44.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73e1ea0d80 20 client.4269.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-08-09T14:50:44.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73e1ea0d80 10 client.4269.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [3,2] primary 3 2024-08-09T14:50:44.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73e1ea0d80 1 --2- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] conn(0x5576d603fe50 0x5576d6040290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:44.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73e1ea0d80 20 client.4269.objecter _get_session s=0x5576d603f7d0 osd=3 3 2024-08-09T14:50:44.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73e1ea0d80 10 client.4269.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.3 2024-08-09T14:50:44.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73e1ea0d80 20 client.4269.objecter get_session s=0x5576d603f7d0 osd=3 3 2024-08-09T14:50:44.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73e1ea0d80 15 client.4269.objecter _session_op_assign 3 1 2024-08-09T14:50:44.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73e1ea0d80 15 client.4269.objecter _send_op 1 to 2.3 on osd.3 2024-08-09T14:50:44.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) -- 0x5576d6064850 con 0x5576d603fe50 2024-08-09T14:50:44.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73e1ea0d80 20 client.4269.objecter put_session s=0x5576d603f7d0 osd=3 4 2024-08-09T14:50:44.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.521+0000 7f73e1ea0d80 5 client.4269.objecter 1 in flight 2024-08-09T14:50:44.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.684+0000 7f73e1bfc640 1 --2- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] conn(0x5576d603fe50 0x5576d6040290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:44.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.692+0000 7f73e1bfc640 10 monclient: get_auth_request con 0x5576d603fe50 auth_method 0 2024-08-09T14:50:44.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.702+0000 7f73d8ff9640 10 client.4269.objecter ms_handle_connect 0x5576d603fe50 2024-08-09T14:50:44.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:44.702+0000 7f73e1bfc640 1 --2- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] conn(0x5576d603fe50 0x5576d6040290 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:46.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.067+0000 7f73e1bfc640 1 -- 172.21.15.155:0/3082486450 <== osd.3 v2:172.21.15.155:6810/3967029756 1 ==== osd_op_reply(1 rbd_trash [create] v24'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7f73d400d2c0 con 0x5576d603fe50 2024-08-09T14:50:46.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.067+0000 7f73e1bfc640 10 client.4269.objecter ms_dispatch 0x5576d5e8bc50 osd_op_reply(1 rbd_trash [create] v24'1 uv1 ondisk = 0) 2024-08-09T14:50:46.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.067+0000 7f73e1bfc640 10 client.4269.objecter in handle_osd_op_reply 2024-08-09T14:50:46.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.067+0000 7f73e1bfc640 7 client.4269.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-08-09T14:50:46.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.067+0000 7f73e1bfc640 10 client.4269.objecter op 0 rval 0 len 0 2024-08-09T14:50:46.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.067+0000 7f73e1bfc640 15 client.4269.objecter handle_osd_op_reply completed tid 1 2024-08-09T14:50:46.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.067+0000 7f73e1bfc640 15 client.4269.objecter _finish_op 1 2024-08-09T14:50:46.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.067+0000 7f73e1bfc640 20 client.4269.objecter put_session s=0x5576d603f7d0 osd=3 4 2024-08-09T14:50:46.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.067+0000 7f73e1bfc640 15 client.4269.objecter _session_op_remove 3 1 2024-08-09T14:50:46.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.067+0000 7f73e1bfc640 5 client.4269.objecter 0 in flight 2024-08-09T14:50:46.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.067+0000 7f73e1ea0d80 10 librados: Objecter returned from create r=0 2024-08-09T14:50:46.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.069+0000 7f73e1ea0d80 10 client.4269.objecter _op_submit op 0x5576d60b6210 2024-08-09T14:50:46.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.069+0000 7f73e1ea0d80 20 client.4269.objecter _calc_target epoch 24 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:50:46.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.069+0000 7f73e1ea0d80 20 client.4269.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-08-09T14:50:46.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.069+0000 7f73e1ea0d80 10 client.4269.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [3,6] primary 3 2024-08-09T14:50:46.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.069+0000 7f73e1ea0d80 20 client.4269.objecter _get_session s=0x5576d603f7d0 osd=3 3 2024-08-09T14:50:46.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.069+0000 7f73e1ea0d80 10 client.4269.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.3 2024-08-09T14:50:46.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.069+0000 7f73e1ea0d80 20 client.4269.objecter get_session s=0x5576d603f7d0 osd=3 3 2024-08-09T14:50:46.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.069+0000 7f73e1ea0d80 15 client.4269.objecter _session_op_assign 3 2 2024-08-09T14:50:46.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.069+0000 7f73e1ea0d80 15 client.4269.objecter _send_op 2 to 2.2 on osd.3 2024-08-09T14:50:46.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.069+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e24) -- 0x5576d60b6760 con 0x5576d603fe50 2024-08-09T14:50:46.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.069+0000 7f73e1ea0d80 20 client.4269.objecter put_session s=0x5576d603f7d0 osd=3 4 2024-08-09T14:50:46.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.069+0000 7f73e1ea0d80 5 client.4269.objecter 1 in flight 2024-08-09T14:50:46.094 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1bfc640 1 -- 172.21.15.155:0/3082486450 <== osd.3 v2:172.21.15.155:6810/3967029756 3 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f73d400d2c0 con 0x5576d603fe50 2024-08-09T14:50:46.095 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1bfc640 10 client.4269.objecter ms_dispatch 0x5576d5e8bc50 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-09T14:50:46.095 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1bfc640 10 client.4269.objecter in handle_osd_op_reply 2024-08-09T14:50:46.095 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1bfc640 7 client.4269.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-08-09T14:50:46.095 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1bfc640 10 client.4269.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-09T14:50:46.095 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1bfc640 10 client.4269.objecter op 0 rval 0 len 0 2024-08-09T14:50:46.095 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1bfc640 15 client.4269.objecter handle_osd_op_reply completed tid 2 2024-08-09T14:50:46.095 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1bfc640 15 client.4269.objecter _finish_op 2 2024-08-09T14:50:46.095 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1bfc640 20 client.4269.objecter put_session s=0x5576d603f7d0 osd=3 4 2024-08-09T14:50:46.095 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1bfc640 15 client.4269.objecter _session_op_remove 3 2 2024-08-09T14:50:46.095 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1bfc640 5 client.4269.objecter 0 in flight 2024-08-09T14:50:46.095 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 <== osd.3 v2:172.21.15.155:6810/3967029756 2 ==== osd_map(25..25 src has 1..25) ==== 288+0+0 (crc 0 0 0) 0x7f73d40019a0 con 0x5576d603fe50 2024-08-09T14:50:46.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73d8ff9640 10 client.4269.objecter ms_dispatch 0x5576d5e8bc50 osd_map(25..25 src has 1..25) 2024-08-09T14:50:46.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73d8ff9640 3 client.4269.objecter handle_osd_map got epochs [25,25] > 24 2024-08-09T14:50:46.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73d8ff9640 3 client.4269.objecter handle_osd_map decoding incremental epoch 25 2024-08-09T14:50:46.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73d8ff9640 20 client.4269.objecter dump_active .. 0 homeless 2024-08-09T14:50:46.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1ea0d80 10 client.4269.objecter _op_submit op 0x5576d60b53a0 2024-08-09T14:50:46.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1ea0d80 20 client.4269.objecter _calc_target epoch 25 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:50:46.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1ea0d80 20 client.4269.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-08-09T14:50:46.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1ea0d80 10 client.4269.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [3,6] primary 3 2024-08-09T14:50:46.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1ea0d80 20 client.4269.objecter _get_session s=0x5576d603f7d0 osd=3 3 2024-08-09T14:50:46.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1ea0d80 10 client.4269.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.3 2024-08-09T14:50:46.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1ea0d80 20 client.4269.objecter get_session s=0x5576d603f7d0 osd=3 3 2024-08-09T14:50:46.096 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1ea0d80 15 client.4269.objecter _session_op_assign 3 3 2024-08-09T14:50:46.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1ea0d80 15 client.4269.objecter _send_op 3 to 2.2 on osd.3 2024-08-09T14:50:46.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e25) -- 0x5576d60b5850 con 0x5576d603fe50 2024-08-09T14:50:46.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1ea0d80 20 client.4269.objecter put_session s=0x5576d603f7d0 osd=3 4 2024-08-09T14:50:46.097 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.093+0000 7f73e1ea0d80 5 client.4269.objecter 1 in flight 2024-08-09T14:50:46.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.099+0000 7f73e1bfc640 1 -- 172.21.15.155:0/3082486450 <== osd.3 v2:172.21.15.155:6810/3967029756 4 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f73d400d2c0 con 0x5576d603fe50 2024-08-09T14:50:46.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.099+0000 7f73e1bfc640 10 client.4269.objecter ms_dispatch 0x5576d5e8bc50 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-09T14:50:46.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.099+0000 7f73e1bfc640 10 client.4269.objecter in handle_osd_op_reply 2024-08-09T14:50:46.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.099+0000 7f73e1bfc640 7 client.4269.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-08-09T14:50:46.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.099+0000 7f73e1bfc640 10 client.4269.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-09T14:50:46.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.099+0000 7f73e1bfc640 10 client.4269.objecter op 0 rval 0 len 0 2024-08-09T14:50:46.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.099+0000 7f73e1bfc640 15 client.4269.objecter handle_osd_op_reply completed tid 3 2024-08-09T14:50:46.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.099+0000 7f73e1bfc640 15 client.4269.objecter _finish_op 3 2024-08-09T14:50:46.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.099+0000 7f73e1bfc640 20 client.4269.objecter put_session s=0x5576d603f7d0 osd=3 4 2024-08-09T14:50:46.101 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.099+0000 7f73e1bfc640 15 client.4269.objecter _session_op_remove 3 3 2024-08-09T14:50:46.102 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.099+0000 7f73e1bfc640 5 client.4269.objecter 0 in flight 2024-08-09T14:50:46.102 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.099+0000 7f73da7fc640 10 client.4269.objecter allocate_selfmanaged_snap; pool: 2 2024-08-09T14:50:46.102 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.100+0000 7f73da7fc640 10 client.4269.objecter pool_op_submit 4 2024-08-09T14:50:46.102 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.100+0000 7f73da7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:46.102 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.100+0000 7f73da7fc640 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f73bc0030d0 con 0x5576d5faa4b0 2024-08-09T14:50:46.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.635+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 <== mon.0 v2:172.21.15.155:3300/0 9 ==== pool_op_reply(tid 4 (0) Success v26) ==== 55+0+0 (secure 0 0 0) 0x7f73d0017b80 con 0x5576d5faa4b0 2024-08-09T14:50:46.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.636+0000 7f73d8ff9640 10 client.4269.objecter ms_dispatch 0x5576d5e8bc50 pool_op_reply(tid 4 (0) Success v26) 2024-08-09T14:50:46.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.636+0000 7f73d8ff9640 10 client.4269.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v26) 2024-08-09T14:50:46.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.636+0000 7f73d8ff9640 10 client.4269.objecter have request 4 at 0x7f73bc003060 Op: create unmanaged snap 2024-08-09T14:50:46.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.636+0000 7f73d8ff9640 20 client.4269.objecter waiting for client to reach epoch 26 before calling back 2024-08-09T14:50:46.637 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.636+0000 7f73d8ff9640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:46.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.636+0000 7f73d8ff9640 10 monclient: _renew_subs 2024-08-09T14:50:46.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.636+0000 7f73d8ff9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:46.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.636+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=26}) -- 0x7f73c0072110 con 0x5576d5faa4b0 2024-08-09T14:50:46.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.636+0000 7f73d8ff9640 10 client.4269.objecter done 2024-08-09T14:50:46.652 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.650+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 <== mon.0 v2:172.21.15.155:3300/0 10 ==== osd_map(26..26 src has 1..26) ==== 626+0+0 (secure 0 0 0) 0x7f73d00a2280 con 0x5576d5faa4b0 2024-08-09T14:50:46.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.650+0000 7f73d8ff9640 10 client.4269.objecter ms_dispatch 0x5576d5e8bc50 osd_map(26..26 src has 1..26) 2024-08-09T14:50:46.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.650+0000 7f73d8ff9640 3 client.4269.objecter handle_osd_map got epochs [26,26] > 25 2024-08-09T14:50:46.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.650+0000 7f73d8ff9640 3 client.4269.objecter handle_osd_map decoding incremental epoch 26 2024-08-09T14:50:46.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.650+0000 7f73d8ff9640 20 client.4269.objecter dump_active .. 0 homeless 2024-08-09T14:50:46.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.651+0000 7f73daffd640 20 librados: queue_aio_write 0x5576d60b17a0 completion 0x5576d5e34ee0 write_seq 1 2024-08-09T14:50:46.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.651+0000 7f73daffd640 10 client.4269.objecter _op_submit op 0x7f73b8006380 2024-08-09T14:50:46.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.651+0000 7f73daffd640 20 client.4269.objecter _calc_target epoch 26 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:50:46.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.651+0000 7f73daffd640 20 client.4269.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-08-09T14:50:46.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.651+0000 7f73daffd640 10 client.4269.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [3,6] primary 3 2024-08-09T14:50:46.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.651+0000 7f73daffd640 20 client.4269.objecter _get_session s=0x5576d603f7d0 osd=3 3 2024-08-09T14:50:46.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.651+0000 7f73daffd640 10 client.4269.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.3 2024-08-09T14:50:46.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.651+0000 7f73daffd640 20 client.4269.objecter get_session s=0x5576d603f7d0 osd=3 3 2024-08-09T14:50:46.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.651+0000 7f73daffd640 15 client.4269.objecter _session_op_assign 3 5 2024-08-09T14:50:46.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.651+0000 7f73daffd640 15 client.4269.objecter _send_op 5 to 2.2 on osd.3 2024-08-09T14:50:46.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.651+0000 7f73daffd640 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e26) -- 0x7f73b8006830 con 0x5576d603fe50 2024-08-09T14:50:46.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.651+0000 7f73daffd640 20 client.4269.objecter put_session s=0x5576d603f7d0 osd=3 4 2024-08-09T14:50:46.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.651+0000 7f73daffd640 5 client.4269.objecter 1 in flight 2024-08-09T14:50:46.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.702+0000 7f73cb7fe640 10 monclient: tick 2024-08-09T14:50:46.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.702+0000 7f73cb7fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:46.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.702+0000 7f73cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:46.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.702+0000 7f73cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:46.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73e1bfc640 1 -- 172.21.15.155:0/3082486450 <== osd.3 v2:172.21.15.155:6810/3967029756 5 ==== osd_op_reply(5 rbd_info [create,write 0~8] v26'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f73d400d2c0 con 0x5576d603fe50 2024-08-09T14:50:46.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73e1bfc640 10 client.4269.objecter ms_dispatch 0x5576d5e8bc50 osd_op_reply(5 rbd_info [create,write 0~8] v26'1 uv1 ondisk = 0) 2024-08-09T14:50:46.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73e1bfc640 10 client.4269.objecter in handle_osd_op_reply 2024-08-09T14:50:46.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73e1bfc640 7 client.4269.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-08-09T14:50:46.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73e1bfc640 10 client.4269.objecter op 0 rval 0 len 0 2024-08-09T14:50:46.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73e1bfc640 10 client.4269.objecter op 1 rval 0 len 0 2024-08-09T14:50:46.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73e1bfc640 15 client.4269.objecter handle_osd_op_reply completed tid 5 2024-08-09T14:50:46.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73e1bfc640 15 client.4269.objecter _finish_op 5 2024-08-09T14:50:46.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73e1bfc640 20 client.4269.objecter put_session s=0x5576d603f7d0 osd=3 4 2024-08-09T14:50:46.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73e1bfc640 15 client.4269.objecter _session_op_remove 3 5 2024-08-09T14:50:46.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73e1bfc640 5 client.4269.objecter 0 in flight 2024-08-09T14:50:46.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73e1bfc640 20 librados: complete_aio_write 0x5576d5e34ee0 2024-08-09T14:50:46.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73da7fc640 10 client.4269.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-08-09T14:50:46.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73da7fc640 10 client.4269.objecter pool_op_submit 6 2024-08-09T14:50:46.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73da7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:46.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:46.873+0000 7f73da7fc640 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v26) -- 0x7f73bc004520 con 0x5576d5faa4b0 2024-08-09T14:50:47.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.702+0000 7f73cb7fe640 10 monclient: tick 2024-08-09T14:50:47.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.702+0000 7f73cb7fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:47.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.702+0000 7f73cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:47.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.705+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 <== mon.0 v2:172.21.15.155:3300/0 11 ==== pool_op_reply(tid 6 (0) Success v27) ==== 43+0+0 (secure 0 0 0) 0x7f73d009a230 con 0x5576d5faa4b0 2024-08-09T14:50:47.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.705+0000 7f73d8ff9640 10 client.4269.objecter ms_dispatch 0x5576d5e8bc50 pool_op_reply(tid 6 (0) Success v27) 2024-08-09T14:50:47.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.705+0000 7f73d8ff9640 10 client.4269.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v27) 2024-08-09T14:50:47.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.705+0000 7f73d8ff9640 10 client.4269.objecter have request 6 at 0x7f73bc003060 Op: delete unmanaged snap 2024-08-09T14:50:47.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.705+0000 7f73d8ff9640 20 client.4269.objecter waiting for client to reach epoch 27 before calling back 2024-08-09T14:50:47.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.705+0000 7f73d8ff9640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:47.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.705+0000 7f73d8ff9640 10 monclient: _renew_subs 2024-08-09T14:50:47.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.705+0000 7f73d8ff9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:47.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.705+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=27}) -- 0x7f73c0072110 con 0x5576d5faa4b0 2024-08-09T14:50:47.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.705+0000 7f73d8ff9640 10 client.4269.objecter done 2024-08-09T14:50:47.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.708+0000 7f73d8ff9640 1 -- 172.21.15.155:0/3082486450 <== mon.0 v2:172.21.15.155:3300/0 12 ==== osd_map(27..27 src has 1..27) ==== 654+0+0 (secure 0 0 0) 0x7f73d00a2480 con 0x5576d5faa4b0 2024-08-09T14:50:47.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.708+0000 7f73d8ff9640 10 client.4269.objecter ms_dispatch 0x5576d5e8bc50 osd_map(27..27 src has 1..27) 2024-08-09T14:50:47.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.708+0000 7f73d8ff9640 3 client.4269.objecter handle_osd_map got epochs [27,27] > 26 2024-08-09T14:50:47.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.708+0000 7f73d8ff9640 3 client.4269.objecter handle_osd_map decoding incremental epoch 27 2024-08-09T14:50:47.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.708+0000 7f73d8ff9640 20 client.4269.objecter dump_active .. 0 homeless 2024-08-09T14:50:47.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.708+0000 7f73daffd640 20 librados: queue_aio_write 0x5576d60b17a0 completion 0x7f73bc001d60 write_seq 2 2024-08-09T14:50:47.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.709+0000 7f73daffd640 10 client.4269.objecter _op_submit op 0x7f73b8006380 2024-08-09T14:50:47.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.709+0000 7f73daffd640 20 client.4269.objecter _calc_target epoch 27 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:50:47.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.709+0000 7f73daffd640 20 client.4269.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-08-09T14:50:47.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.709+0000 7f73daffd640 10 client.4269.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [3,6] primary 3 2024-08-09T14:50:47.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.709+0000 7f73daffd640 20 client.4269.objecter _get_session s=0x5576d603f7d0 osd=3 3 2024-08-09T14:50:47.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.709+0000 7f73daffd640 10 client.4269.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.3 2024-08-09T14:50:47.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.709+0000 7f73daffd640 20 client.4269.objecter get_session s=0x5576d603f7d0 osd=3 3 2024-08-09T14:50:47.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.709+0000 7f73daffd640 15 client.4269.objecter _session_op_assign 3 7 2024-08-09T14:50:47.712 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.709+0000 7f73daffd640 15 client.4269.objecter _send_op 7 to 2.2 on osd.3 2024-08-09T14:50:47.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.709+0000 7f73daffd640 1 -- 172.21.15.155:0/3082486450 --> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27) -- 0x7f73b8006bd0 con 0x5576d603fe50 2024-08-09T14:50:47.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.709+0000 7f73daffd640 20 client.4269.objecter put_session s=0x5576d603f7d0 osd=3 4 2024-08-09T14:50:47.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.709+0000 7f73daffd640 5 client.4269.objecter 1 in flight 2024-08-09T14:50:47.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1bfc640 1 -- 172.21.15.155:0/3082486450 <== osd.3 v2:172.21.15.155:6810/3967029756 6 ==== osd_op_reply(7 rbd_info [write 0~19] v27'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f73d400d2c0 con 0x5576d603fe50 2024-08-09T14:50:47.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1bfc640 10 client.4269.objecter ms_dispatch 0x5576d5e8bc50 osd_op_reply(7 rbd_info [write 0~19] v27'2 uv2 ondisk = 0) 2024-08-09T14:50:47.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1bfc640 10 client.4269.objecter in handle_osd_op_reply 2024-08-09T14:50:47.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1bfc640 7 client.4269.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-08-09T14:50:47.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1bfc640 10 client.4269.objecter op 0 rval 0 len 0 2024-08-09T14:50:47.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1bfc640 15 client.4269.objecter handle_osd_op_reply completed tid 7 2024-08-09T14:50:47.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1bfc640 15 client.4269.objecter _finish_op 7 2024-08-09T14:50:47.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1bfc640 20 client.4269.objecter put_session s=0x5576d603f7d0 osd=3 4 2024-08-09T14:50:47.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1bfc640 15 client.4269.objecter _session_op_remove 3 7 2024-08-09T14:50:47.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1bfc640 5 client.4269.objecter 0 in flight 2024-08-09T14:50:47.944 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1bfc640 20 librados: complete_aio_write 0x7f73bc001d60 2024-08-09T14:50:47.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 10 librados: watch_flush enter 2024-08-09T14:50:47.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 10 librados: watch_flush exit 2024-08-09T14:50:47.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 10 client.4269.objecter close_session for osd.3 2024-08-09T14:50:47.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] conn(0x5576d603fe50 msgr2=0x5576d6040290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:47.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 1 --2- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] conn(0x5576d603fe50 0x5576d6040290 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:47.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 20 client.4269.objecter put_session s=0x5576d603f7d0 osd=3 1 2024-08-09T14:50:47.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 20 client.4269.objecter shutdown clearing up homeless session... 2024-08-09T14:50:47.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 10 client.4269.objecter successfully canceled tick 2024-08-09T14:50:47.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f73c006b940 msgr2=0x7f73c006de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:47.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 1 --2- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f73c006b940 0x7f73c006de00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f73cc006fd0 tx=0x7f73cc0069a0 comp rx=0 tx=0).stop 2024-08-09T14:50:47.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 10 monclient: shutdown 2024-08-09T14:50:47.945 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:47.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576d5faa4b0 msgr2=0x5576d60417d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:47.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 1 --2- 172.21.15.155:0/3082486450 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x5576d5faa4b0 0x5576d60417d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f73d000a4f0 tx=0x7f73d0002fb0 comp rx=0 tx=0).stop 2024-08-09T14:50:47.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1bfc640 1 -- 172.21.15.155:0/3082486450 reap_dead start 2024-08-09T14:50:47.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/3082486450 shutdown_connections 2024-08-09T14:50:47.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/3082486450 >> 172.21.15.155:0/3082486450 conn(0x5576d5f88d20 msgr2=0x5576d5fb4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:47.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/3082486450 shutdown_connections 2024-08-09T14:50:47.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 1 -- 172.21.15.155:0/3082486450 wait complete. 2024-08-09T14:50:47.946 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:47.941+0000 7f73e1ea0d80 1 librados: shutdown 2024-08-09T14:50:47.947 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-08-09T14:50:47.948 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-09T14:50:47.948 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-08-09T14:50:47.948 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-08-09T14:50:47.949 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-08-09T14:50:48.048 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.046+0000 7fbec36c4640 10 monclient: get_monmap_and_config 2024-08-09T14:50:48.048 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.046+0000 7fbec36c4640 10 monclient: build_initial_monmap 2024-08-09T14:50:48.048 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.047+0000 7fbec36c4640 10 monclient: monmap: 2024-08-09T14:50:48.048 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:48.048 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:48.048 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:48.048025+0000 2024-08-09T14:50:48.048 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:48.048025+0000 2024-08-09T14:50:48.049 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:48.049 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:48.049 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:48.049 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:48.049 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:48.049 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:48.049 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:48.049 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:48.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.047+0000 7fbec36c4640 1 Processor -- start 2024-08-09T14:50:48.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.047+0000 7fbec36c4640 1 -- start start 2024-08-09T14:50:48.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.047+0000 7fbec36c4640 10 monclient: init 2024-08-09T14:50:48.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:48.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-08-09T14:50:48.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fbebc06ce20 0x7fbebc06d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:48.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 10 monclient(hunting): picked mon.noname-c con 0x7fbebc06ce20 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:48.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbebc06e010 0x7fbebc13ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:48.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 10 monclient(hunting): picked mon.noname-a con 0x7fbebc06e010 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:48.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbebc140520 0x7fbebc0b1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:48.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 10 monclient(hunting): picked mon.noname-b con 0x7fbebc140520 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:48.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:48.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7fbebc146980 con 0x7fbebc06e010 2024-08-09T14:50:48.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:48.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7fbebc146ea0 con 0x7fbebc06ce20 2024-08-09T14:50:48.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:48.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fbebc147050 con 0x7fbebc140520 2024-08-09T14:50:48.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:48.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.048+0000 7fbec36c4640 10 monclient(hunting): authenticate will time out at 1255.085496s 2024-08-09T14:50:48.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec0c5f640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fbebc06e010 0x7fbebc13ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:48.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec1c61640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbebc140520 0x7fbebc0b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:48.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec1460640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fbebc06ce20 0x7fbebc06d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:48.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec1c61640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbebc140520 0x7fbebc0b1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54560/0 (socket says 172.21.15.155:54560) 2024-08-09T14:50:48.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec1c61640 1 -- 172.21.15.155:0/1522 learned_addr learned my addr 172.21.15.155:0/1522 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:48.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec1c61640 10 monclient(hunting): get_auth_request con 0x7fbebc140520 auth_method 0 2024-08-09T14:50:48.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec1c61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:48.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec1c61640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:48.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec1c61640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:48.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec1460640 10 monclient(hunting): get_auth_request con 0x7fbebc06ce20 auth_method 0 2024-08-09T14:50:48.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec1460640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:48.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec1460640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:48.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec1460640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:48.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec0c5f640 10 monclient(hunting): get_auth_request con 0x7fbebc06e010 auth_method 0 2024-08-09T14:50:48.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec0c5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:48.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec0c5f640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:48.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec0c5f640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:48.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec1c61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:48.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.049+0000 7fbec1c61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:48.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.050+0000 7fbec1c61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:48.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.050+0000 7fbec0c5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:48.054 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.050+0000 7fbec0c5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:48.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.050+0000 7fbec0c5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:48.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.050+0000 7fbec1460640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:48.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.050+0000 7fbec1460640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:48.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.050+0000 7fbec1460640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:48.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.051+0000 7fbec1c61640 10 monclient(hunting): handle_auth_done global_id 4247 payload 274 2024-08-09T14:50:48.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.051+0000 7fbec1c61640 1 -- 172.21.15.155:0/1522 >> v2:172.21.15.194:3300/0 conn(0x7fbebc06ce20 msgr2=0x7fbebc06d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:48.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.051+0000 7fbec1c61640 1 --2- 172.21.15.155:0/1522 >> v2:172.21.15.194:3300/0 conn(0x7fbebc06ce20 0x7fbebc06d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.051+0000 7fbec1c61640 1 -- 172.21.15.155:0/1522 >> v2:172.21.15.155:3300/0 conn(0x7fbebc06e010 msgr2=0x7fbebc13ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:48.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.051+0000 7fbec1c61640 1 --2- 172.21.15.155:0/1522 >> v2:172.21.15.155:3300/0 conn(0x7fbebc06e010 0x7fbebc13ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.051+0000 7fbec1c61640 10 monclient: _finish_hunting 0 2024-08-09T14:50:48.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.051+0000 7fbec1c61640 1 monclient: found mon.noname-b 2024-08-09T14:50:48.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.051+0000 7fbec1c61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:48.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.051+0000 7fbec1c61640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:48.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.051+0000 7fbec1c61640 1 -- 172.21.15.155:0/1522 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbebc146570 con 0x7fbebc140520 2024-08-09T14:50:48.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.054+0000 7fbec1c61640 1 --2- 172.21.15.155:0/1522 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbebc140520 0x7fbebc0b1150 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009e50 tx=0x7fbeb402be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29949e1691dbe3f4 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:48.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.057+0000 7fbeaffff640 1 -- 172.21.15.155:0/1522 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb402c9e0 con 0x7fbebc140520 2024-08-09T14:50:48.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.057+0000 7fbeaffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:48.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbeaffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:48.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbeaffff640 10 monclient: dump: 2024-08-09T14:50:48.059 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:48.059 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:48.059 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.059 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.059 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:48.059 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:48.060 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:48.060 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:48.060 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:48.060 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:48.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbeaffff640 10 monclient: _finish_auth 0 2024-08-09T14:50:48.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbeaffff640 10 monclient: _check_auth_tickets 2024-08-09T14:50:48.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbeaffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:48.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbeaffff640 1 -- 172.21.15.155:0/1522 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeb402cb80 con 0x7fbebc140520 2024-08-09T14:50:48.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbeaffff640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:48.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbeaffff640 1 -- 172.21.15.155:0/1522 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb40343f0 con 0x7fbebc140520 2024-08-09T14:50:48.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbec36c4640 5 monclient: authenticate success, global_id 4247 2024-08-09T14:50:48.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbec36c4640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:48.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbec36c4640 4 set_mon_vals no callback set 2024-08-09T14:50:48.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbeaffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:48.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbeaffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:48.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbeaffff640 10 monclient: dump: 2024-08-09T14:50:48.061 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:48.061 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:48.061 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.061 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.061 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:48.061 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:48.061 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:48.061 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:48.061 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:48.061 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:48.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbec36c4640 10 monclient: shutdown 2024-08-09T14:50:48.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbec36c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:48.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbec36c4640 1 -- 172.21.15.155:0/1522 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbebc140520 msgr2=0x7fbebc0b1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:48.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbec36c4640 1 --2- 172.21.15.155:0/1522 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbebc140520 0x7fbebc0b1150 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4009e50 tx=0x7fbeb402be40 comp rx=0 tx=0).stop 2024-08-09T14:50:48.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbec36c4640 1 -- 172.21.15.155:0/1522 shutdown_connections 2024-08-09T14:50:48.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbec36c4640 1 --2- 172.21.15.155:0/1522 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbebc140520 0x7fbebc0b1150 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbec36c4640 1 --2- 172.21.15.155:0/1522 >> v2:172.21.15.155:3300/0 conn(0x7fbebc06e010 0x7fbebc13ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbec36c4640 1 --2- 172.21.15.155:0/1522 >> v2:172.21.15.194:3300/0 conn(0x7fbebc06ce20 0x7fbebc06d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbec36c4640 1 -- 172.21.15.155:0/1522 >> 172.21.15.155:0/1522 conn(0x7fbebc068030 msgr2=0x7fbebc0b6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:48.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbec36c4640 1 -- 172.21.15.155:0/1522 shutdown_connections 2024-08-09T14:50:48.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.058+0000 7fbec36c4640 1 -- 172.21.15.155:0/1522 wait complete. 2024-08-09T14:50:48.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.059+0000 7fbec36c4640 10 monclient: build_initial_monmap 2024-08-09T14:50:48.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.059+0000 7fbec36c4640 10 monclient: monmap: 2024-08-09T14:50:48.066 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:48.066 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:48.066 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:48.066 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:48.066 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:48.066 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:48.066 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:48.066 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:48.066 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:48.067 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:48.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 1 librados: starting msgr at 2024-08-09T14:50:48.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 1 librados: starting objecter 2024-08-09T14:50:48.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 1 Processor -- start 2024-08-09T14:50:48.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 1 -- start start 2024-08-09T14:50:48.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 1 librados: setting wanted keys 2024-08-09T14:50:48.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 1 librados: calling monclient init 2024-08-09T14:50:48.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 10 monclient: init 2024-08-09T14:50:48.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:48.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-08-09T14:50:48.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbebc06ce20 0x7fbebc13bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:48.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 10 monclient(hunting): picked mon.noname-a con 0x7fbebc06ce20 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:48.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbebc06e010 0x7fbebc1ce2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:48.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 10 monclient(hunting): picked mon.noname-b con 0x7fbebc06e010 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:48.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbebc140520 0x7fbebc1ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:48.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 10 monclient(hunting): picked mon.noname-c con 0x7fbebc140520 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:48.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:48.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fbebc0b1a20 con 0x7fbebc06ce20 2024-08-09T14:50:48.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:48.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbebc0b18a0 con 0x7fbebc06e010 2024-08-09T14:50:48.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:48.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fbebc142bb0 con 0x7fbebc140520 2024-08-09T14:50:48.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:48.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.060+0000 7fbec36c4640 10 monclient(hunting): authenticate will time out at 1255.097878s 2024-08-09T14:50:48.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec1c61640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbebc140520 0x7fbebc1ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:48.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec1460640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbebc06ce20 0x7fbebc13bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:48.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec0c5f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbebc06e010 0x7fbebc1ce2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:48.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec1c61640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbebc140520 0x7fbebc1ce7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54568/0 (socket says 172.21.15.155:54568) 2024-08-09T14:50:48.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec1c61640 1 -- 172.21.15.155:0/2875412110 learned_addr learned my addr 172.21.15.155:0/2875412110 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:48.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec1c61640 10 monclient(hunting): get_auth_request con 0x7fbebc140520 auth_method 0 2024-08-09T14:50:48.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec1c61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:48.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec1c61640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:48.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec1c61640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:48.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec0c5f640 10 monclient(hunting): get_auth_request con 0x7fbebc06e010 auth_method 0 2024-08-09T14:50:48.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec0c5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:48.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec0c5f640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:48.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec0c5f640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:48.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec1460640 10 monclient(hunting): get_auth_request con 0x7fbebc06ce20 auth_method 0 2024-08-09T14:50:48.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec1460640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:48.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec1460640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:48.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.061+0000 7fbec1460640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:48.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.062+0000 7fbec1c61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:48.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.062+0000 7fbec1c61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:48.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.062+0000 7fbec1c61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:48.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.062+0000 7fbec0c5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:48.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.062+0000 7fbec0c5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:48.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.062+0000 7fbec0c5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:48.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.062+0000 7fbec1460640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:48.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.062+0000 7fbec1460640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:48.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.062+0000 7fbec1460640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:48.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.064+0000 7fbec0c5f640 10 monclient(hunting): handle_auth_done global_id 4249 payload 824 2024-08-09T14:50:48.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.064+0000 7fbec0c5f640 1 -- 172.21.15.155:0/2875412110 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbebc140520 msgr2=0x7fbebc1ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:48.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.064+0000 7fbec0c5f640 1 --2- 172.21.15.155:0/2875412110 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbebc140520 0x7fbebc1ce7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.064+0000 7fbec0c5f640 1 -- 172.21.15.155:0/2875412110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbebc06ce20 msgr2=0x7fbebc13bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:48.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.064+0000 7fbec0c5f640 1 --2- 172.21.15.155:0/2875412110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbebc06ce20 0x7fbebc13bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.064+0000 7fbec0c5f640 10 monclient: _finish_hunting 0 2024-08-09T14:50:48.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.064+0000 7fbec0c5f640 1 monclient: found mon.noname-b 2024-08-09T14:50:48.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.064+0000 7fbec0c5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:48.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.064+0000 7fbec0c5f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:48.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.064+0000 7fbec0c5f640 1 -- 172.21.15.155:0/2875412110 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbebc1cdc20 con 0x7fbebc06e010 2024-08-09T14:50:48.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.067+0000 7fbec0c5f640 1 --2- 172.21.15.155:0/2875412110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbebc06e010 0x7fbebc1ce2b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000e850 tx=0x7fbeb000ed10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:48.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.067+0000 7fbeadffb640 10 client.?.objecter ms_handle_connect 0x7fbebc06e010 2024-08-09T14:50:48.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.067+0000 7fbeadffb640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:48.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbeadffb640 1 -- 172.21.15.155:0/2875412110 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb0017070 con 0x7fbebc06e010 2024-08-09T14:50:48.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbeadffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:48.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbeadffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:48.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbeadffb640 10 monclient: dump: 2024-08-09T14:50:48.075 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:48.075 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:48.075 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.075 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.075 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:48.075 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:48.075 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:48.075 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:48.075 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:48.075 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:48.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbeadffb640 10 monclient: _finish_auth 0 2024-08-09T14:50:48.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbeadffb640 10 monclient: _check_auth_tickets 2024-08-09T14:50:48.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbeadffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:48.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbeadffb640 1 -- 172.21.15.155:0/2875412110 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbeb000c940 con 0x7fbebc06e010 2024-08-09T14:50:48.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbeadffb640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:48.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbeadffb640 1 -- 172.21.15.155:0/2875412110 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeb000cd10 con 0x7fbebc06e010 2024-08-09T14:50:48.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbeadffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:48.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbeadffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:48.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbeadffb640 10 monclient: dump: 2024-08-09T14:50:48.076 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:48.076 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:48.077 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.077 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.077 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:48.077 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:48.077 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:48.077 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:48.077 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:48.077 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:48.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbec36c4640 5 monclient: authenticate success, global_id 4249 2024-08-09T14:50:48.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbec36c4640 10 monclient: _renew_subs 2024-08-09T14:50:48.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbec36c4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:48.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbec36c4640 1 -- 172.21.15.155:0/2875412110 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbebc1ccd70 con 0x7fbebc06e010 2024-08-09T14:50:48.077 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.072+0000 7fbeaf7fe640 4 set_mon_vals no callback set 2024-08-09T14:50:48.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.073+0000 7fbec36c4640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:48.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.073+0000 7fbec36c4640 10 monclient: _renew_subs 2024-08-09T14:50:48.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.073+0000 7fbec36c4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:48.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.073+0000 7fbec36c4640 1 -- 172.21.15.155:0/2875412110 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbebc1ced30 con 0x7fbebc06e010 2024-08-09T14:50:48.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.073+0000 7fbec36c4640 1 librados: init done 2024-08-09T14:50:48.078 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.073+0000 7fbec36c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:48.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.073+0000 7fbec36c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:48.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.073+0000 7fbec36c4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:48.079 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.073+0000 7fbec36c4640 1 -- 172.21.15.155:0/2875412110 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbebc13ffe0 con 0x7fbebc06e010 2024-08-09T14:50:48.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.335+0000 7fbeadffb640 1 -- 172.21.15.155:0/2875412110 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fbeb001a070 con 0x7fbebc06e010 2024-08-09T14:50:48.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.336+0000 7fbeadffb640 1 --2- 172.21.15.155:0/2875412110 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fbea006d810 0x7fbea006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:48.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.336+0000 7fbec1460640 1 --2- 172.21.15.155:0/2875412110 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fbea006d810 0x7fbea006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:48.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.336+0000 7fbec1460640 10 monclient: get_auth_request con 0x7fbea006d810 auth_method 0 2024-08-09T14:50:48.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.336+0000 7fbeadffb640 10 client.4249.objecter ms_handle_connect 0x7fbea006d810 2024-08-09T14:50:48.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.336+0000 7fbec1460640 1 --2- 172.21.15.155:0/2875412110 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fbea006d810 0x7fbea006fcd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009b40 tx=0x7fbeb8009190 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:48.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.340+0000 7fbeadffb640 1 -- 172.21.15.155:0/2875412110 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7fbeb0057910 con 0x7fbebc06e010 2024-08-09T14:50:48.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.340+0000 7fbeadffb640 10 client.4249.objecter ms_dispatch 0x7fbebc000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:48.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.340+0000 7fbeadffb640 3 client.4249.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:48.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.340+0000 7fbeadffb640 3 client.4249.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:48.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.340+0000 7fbeadffb640 20 client.4249.objecter dump_active .. 0 homeless 2024-08-09T14:50:48.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.351+0000 7fbeadffb640 1 -- 172.21.15.155:0/2875412110 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7fbeb0059870 con 0x7fbebc06e010 2024-08-09T14:50:48.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.351+0000 7fbeadffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:48.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.351+0000 7fbeadffb640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:48.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.584+0000 7fbec36c4640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-08-09T14:50:48.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.584+0000 7fbec36c4640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-08-09T14:50:48.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.584+0000 7fbec36c4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:48.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.584+0000 7fbec36c4640 1 -- 172.21.15.155:0/2875412110 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fbebc06dd50 con 0x7fbebc06e010 2024-08-09T14:50:48.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.761+0000 7fbeadffb640 1 -- 172.21.15.155:0/2875412110 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+172943 (secure 0 0 0) 0x7fbeb005db70 con 0x7fbebc06e010 2024-08-09T14:50:48.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.761+0000 7fbeadffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-08-09T14:50:48.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.761+0000 7fbeadffb640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:48.765 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:50:48.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 10 librados: watch_flush enter 2024-08-09T14:50:48.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 10 librados: watch_flush exit 2024-08-09T14:50:48.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 20 client.4249.objecter shutdown clearing up homeless session... 2024-08-09T14:50:48.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 10 client.4249.objecter successfully canceled tick 2024-08-09T14:50:48.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 1 -- 172.21.15.155:0/2875412110 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fbea006d810 msgr2=0x7fbea006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:48.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 1 --2- 172.21.15.155:0/2875412110 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fbea006d810 0x7fbea006fcd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fbeb8009b40 tx=0x7fbeb8009190 comp rx=0 tx=0).stop 2024-08-09T14:50:48.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 10 monclient: shutdown 2024-08-09T14:50:48.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:48.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 1 -- 172.21.15.155:0/2875412110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbebc06e010 msgr2=0x7fbebc1ce2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:48.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 1 --2- 172.21.15.155:0/2875412110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbebc06e010 0x7fbebc1ce2b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000e850 tx=0x7fbeb000ed10 comp rx=0 tx=0).stop 2024-08-09T14:50:48.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 1 -- 172.21.15.155:0/2875412110 shutdown_connections 2024-08-09T14:50:48.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 1 --2- 172.21.15.155:0/2875412110 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fbea006d810 0x7fbea006fcd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 1 --2- 172.21.15.155:0/2875412110 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fbebc140520 0x7fbebc1ce7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 1 --2- 172.21.15.155:0/2875412110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbebc06e010 0x7fbebc1ce2b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 1 --2- 172.21.15.155:0/2875412110 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fbebc06ce20 0x7fbebc13bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 1 -- 172.21.15.155:0/2875412110 >> 172.21.15.155:0/2875412110 conn(0x7fbebc068030 msgr2=0x7fbebc142fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:48.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 1 -- 172.21.15.155:0/2875412110 shutdown_connections 2024-08-09T14:50:48.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 1 -- 172.21.15.155:0/2875412110 wait complete. 2024-08-09T14:50:48.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.764+0000 7fbec36c4640 1 librados: shutdown 2024-08-09T14:50:48.777 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6800","nonce":689799774},{"type":"v1","addr":"172.21.15.155:6801","nonce":689799774}]},"active_addr":"172.21.15.155:6801/689799774","active_change":"2024-08-09T14:49:47.052123+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4110,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.155:0","nonce":2731312368}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.155:0","nonce":2653744869}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.155:0","nonce":2279018890}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.155:0","nonce":1536135937}]}]} 2024-08-09T14:50:48.781 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-08-09T14:50:48.781 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-08-09T14:50:48.782 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-09T14:50:48.884 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.883+0000 7fb2b0ce9640 10 monclient: get_monmap_and_config 2024-08-09T14:50:48.884 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.883+0000 7fb2b0ce9640 10 monclient: build_initial_monmap 2024-08-09T14:50:48.885 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.883+0000 7fb2b0ce9640 10 monclient: monmap: 2024-08-09T14:50:48.885 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:48.885 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:48.885 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:48.884318+0000 2024-08-09T14:50:48.885 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:48.884318+0000 2024-08-09T14:50:48.885 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:48.885 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:48.885 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:48.885 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:48.885 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:48.885 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:48.885 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:48.885 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:48.888 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 1 Processor -- start 2024-08-09T14:50:48.888 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 1 -- start start 2024-08-09T14:50:48.888 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 10 monclient: init 2024-08-09T14:50:48.888 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:48.888 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-08-09T14:50:48.888 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb2ac0b0160 0x7fb2ac0b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:48.888 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 10 monclient(hunting): picked mon.noname-a con 0x7fb2ac0b0160 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:48.888 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb2ac0b22e0 0x7fb2ac1469d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:48.888 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 10 monclient(hunting): picked mon.noname-c con 0x7fb2ac0b22e0 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:48.888 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb2ac146f10 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:48.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2ac1461c0 con 0x7fb2ac146f10 2024-08-09T14:50:48.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:48.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7fb2ac0b26e0 con 0x7fb2ac0b0160 2024-08-09T14:50:48.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:48.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7fb2ac0b2860 con 0x7fb2ac0b22e0 2024-08-09T14:50:48.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:48.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.884+0000 7fb2b0ce9640 10 monclient(hunting): authenticate will time out at 1255.921817s 2024-08-09T14:50:48.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.885+0000 7fb2aa59c640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb2ac0b0160 0x7fb2ac0b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:48.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.885+0000 7fb2a3fff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb2ac0b22e0 0x7fb2ac1469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:48.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.885+0000 7fb2aa59c640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fb2ac0b0160 0x7fb2ac0b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43408/0 (socket says 172.21.15.155:43408) 2024-08-09T14:50:48.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.885+0000 7fb2aa59c640 1 -- 172.21.15.155:0/4292593849 learned_addr learned my addr 172.21.15.155:0/4292593849 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:48.889 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.885+0000 7fb2aa59c640 10 monclient(hunting): get_auth_request con 0x7fb2ac0b0160 auth_method 0 2024-08-09T14:50:48.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.885+0000 7fb2aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:48.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.885+0000 7fb2aa59c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:48.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.885+0000 7fb2aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:48.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.886+0000 7fb2a3fff640 10 monclient(hunting): get_auth_request con 0x7fb2ac0b22e0 auth_method 0 2024-08-09T14:50:48.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.886+0000 7fb2a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:48.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.886+0000 7fb2a3fff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:48.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.886+0000 7fb2a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:48.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.886+0000 7fb2aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:48.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.886+0000 7fb2aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:48.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.886+0000 7fb2aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:48.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.886+0000 7fb2a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:48.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.886+0000 7fb2a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:48.890 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.886+0000 7fb2a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:48.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.887+0000 7fb2aa59c640 10 monclient(hunting): handle_auth_done global_id 4278 payload 274 2024-08-09T14:50:48.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.887+0000 7fb2aa59c640 1 -- 172.21.15.155:0/4292593849 >> v2:172.21.15.194:3300/0 conn(0x7fb2ac0b22e0 msgr2=0x7fb2ac1469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:48.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.887+0000 7fb2aa59c640 1 --2- 172.21.15.155:0/4292593849 >> v2:172.21.15.194:3300/0 conn(0x7fb2ac0b22e0 0x7fb2ac1469d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.887+0000 7fb2aa59c640 1 -- 172.21.15.155:0/4292593849 >> v1:172.21.15.194:6789/0 conn(0x7fb2ac146f10 legacy=0x7fb2ac0b3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:48.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.887+0000 7fb2aa59c640 10 monclient: _finish_hunting 0 2024-08-09T14:50:48.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.887+0000 7fb2aa59c640 1 monclient: found mon.noname-a 2024-08-09T14:50:48.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.887+0000 7fb2aa59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:48.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.887+0000 7fb2aa59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:48.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.887+0000 7fb2aa59c640 1 -- 172.21.15.155:0/4292593849 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2ac0b3f50 con 0x7fb2ac0b0160 2024-08-09T14:50:48.892 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.889+0000 7fb2aa59c640 1 --2- 172.21.15.155:0/4292593849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2ac0b0160 0x7fb2ac0b0540 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac0b6460 tx=0x7fb29c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f77a7c573714394 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:48.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2a9d9b640 1 -- 172.21.15.155:0/4292593849 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c02e070 con 0x7fb2ac0b0160 2024-08-09T14:50:48.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2a9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2a9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2a9d9b640 10 monclient: dump: 2024-08-09T14:50:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:48.895 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:48.895 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:48.896 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2a9d9b640 10 monclient: _finish_auth 0 2024-08-09T14:50:48.896 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2a9d9b640 10 monclient: _check_auth_tickets 2024-08-09T14:50:48.896 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2a9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:48.896 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2a9d9b640 1 -- 172.21.15.155:0/4292593849 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb29c037070 con 0x7fb2ac0b0160 2024-08-09T14:50:48.896 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2b0ce9640 5 monclient: authenticate success, global_id 4278 2024-08-09T14:50:48.897 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2b0ce9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:48.897 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2a9d9b640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:48.897 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2b0ce9640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:48.897 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2b0ce9640 4 set_mon_vals no callback set 2024-08-09T14:50:48.897 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2a9d9b640 1 -- 172.21.15.155:0/4292593849 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb29c031a00 con 0x7fb2ac0b0160 2024-08-09T14:50:48.898 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2a9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:48.898 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2a9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:48.898 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.893+0000 7fb2a9d9b640 10 monclient: dump: 2024-08-09T14:50:48.898 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:48.898 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:48.898 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.899 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.899 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:48.899 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:48.899 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:48.899 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:48.899 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:48.899 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:48.905 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.894+0000 7fb2b0ce9640 10 monclient: shutdown 2024-08-09T14:50:48.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.894+0000 7fb2b0ce9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:48.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.894+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/4292593849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2ac0b0160 msgr2=0x7fb2ac0b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:48.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.894+0000 7fb2b0ce9640 1 --2- 172.21.15.155:0/4292593849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2ac0b0160 0x7fb2ac0b0540 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac0b6460 tx=0x7fb29c02b880 comp rx=0 tx=0).stop 2024-08-09T14:50:48.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.894+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/4292593849 shutdown_connections 2024-08-09T14:50:48.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.894+0000 7fb2b0ce9640 1 --2- 172.21.15.155:0/4292593849 >> v2:172.21.15.194:3300/0 conn(0x7fb2ac0b22e0 0x7fb2ac1469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.894+0000 7fb2b0ce9640 1 --2- 172.21.15.155:0/4292593849 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2ac0b0160 0x7fb2ac0b0540 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.894+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/4292593849 >> 172.21.15.155:0/4292593849 conn(0x7fb2ac06dbd0 msgr2=0x7fb2ac0676f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:48.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.894+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/4292593849 shutdown_connections 2024-08-09T14:50:48.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.894+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/4292593849 wait complete. 2024-08-09T14:50:48.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.895+0000 7fb2b0ce9640 10 monclient: build_initial_monmap 2024-08-09T14:50:48.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.895+0000 7fb2b0ce9640 10 monclient: monmap: 2024-08-09T14:50:48.907 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:48.907 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:48.907 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:48.907 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:48.907 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:48.907 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:48.907 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:48.907 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:48.907 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:48.907 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:48.907 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.895+0000 7fb2b0ce9640 1 librados: starting msgr at 2024-08-09T14:50:48.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.895+0000 7fb2b0ce9640 1 librados: starting objecter 2024-08-09T14:50:48.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.895+0000 7fb2b0ce9640 1 Processor -- start 2024-08-09T14:50:48.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.895+0000 7fb2b0ce9640 1 -- start start 2024-08-09T14:50:48.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.895+0000 7fb2b0ce9640 1 librados: setting wanted keys 2024-08-09T14:50:48.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.895+0000 7fb2b0ce9640 1 librados: calling monclient init 2024-08-09T14:50:48.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.895+0000 7fb2b0ce9640 10 monclient: init 2024-08-09T14:50:48.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:48.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 10 monclient: _add_conns ranks=[0,2,1] 2024-08-09T14:50:48.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2ac0b0160 0x7fb2ac14c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:48.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 10 monclient(hunting): picked mon.noname-a con 0x7fb2ac0b0160 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:48.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2ac0b22e0 0x7fb2ac0aa490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:48.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 10 monclient(hunting): picked mon.noname-c con 0x7fb2ac0b22e0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:48.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2ac146f10 0x7fb2ac0aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:48.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 10 monclient(hunting): picked mon.noname-b con 0x7fb2ac146f10 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:48.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:48.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac0b42d0 con 0x7fb2ac0b0160 2024-08-09T14:50:48.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:48.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb2ac0b29e0 con 0x7fb2ac146f10 2024-08-09T14:50:48.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:48.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fb2ac14d760 con 0x7fb2ac0b22e0 2024-08-09T14:50:48.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:48.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2b0ce9640 10 monclient(hunting): authenticate will time out at 1255.933472s 2024-08-09T14:50:48.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2aa59c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2ac0b0160 0x7fb2ac14c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:48.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.896+0000 7fb2a3fff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2ac0b22e0 0x7fb2ac0aa490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:48.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aad9d640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2ac146f10 0x7fb2ac0aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:48.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aa59c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2ac0b0160 0x7fb2ac14c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43418/0 (socket says 172.21.15.155:43418) 2024-08-09T14:50:48.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aa59c640 1 -- 172.21.15.155:0/2799548861 learned_addr learned my addr 172.21.15.155:0/2799548861 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:48.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2a3fff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2ac0b22e0 0x7fb2ac0aa490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54572/0 (socket says 172.21.15.155:54572) 2024-08-09T14:50:48.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aa59c640 10 monclient(hunting): get_auth_request con 0x7fb2ac0b0160 auth_method 0 2024-08-09T14:50:48.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:48.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aa59c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:48.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:48.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2a3fff640 10 monclient(hunting): get_auth_request con 0x7fb2ac0b22e0 auth_method 0 2024-08-09T14:50:48.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:48.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2a3fff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:48.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:48.911 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aad9d640 10 monclient(hunting): get_auth_request con 0x7fb2ac146f10 auth_method 0 2024-08-09T14:50:48.911 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:48.911 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aad9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:48.911 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:48.911 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:48.911 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:48.911 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:48.911 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:48.911 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:48.911 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:48.911 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:48.911 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:48.911 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.897+0000 7fb2aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:48.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.900+0000 7fb2aad9d640 10 monclient(hunting): handle_auth_done global_id 4255 payload 824 2024-08-09T14:50:48.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.900+0000 7fb2aad9d640 1 -- 172.21.15.155:0/2799548861 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2ac0b22e0 msgr2=0x7fb2ac0aa490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:48.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.900+0000 7fb2aad9d640 1 --2- 172.21.15.155:0/2799548861 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2ac0b22e0 0x7fb2ac0aa490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.900+0000 7fb2aad9d640 1 -- 172.21.15.155:0/2799548861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2ac0b0160 msgr2=0x7fb2ac14c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:48.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.900+0000 7fb2aad9d640 1 --2- 172.21.15.155:0/2799548861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2ac0b0160 0x7fb2ac14c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:48.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.900+0000 7fb2aad9d640 10 monclient: _finish_hunting 0 2024-08-09T14:50:48.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.900+0000 7fb2aad9d640 1 monclient: found mon.noname-b 2024-08-09T14:50:48.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.900+0000 7fb2aad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:48.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.900+0000 7fb2aad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:48.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.900+0000 7fb2aad9d640 1 -- 172.21.15.155:0/2799548861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2ac0abb70 con 0x7fb2ac146f10 2024-08-09T14:50:48.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.900+0000 7fb2aa59c640 1 --2- 172.21.15.155:0/2799548861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2ac0b0160 0x7fb2ac14c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-08-09T14:50:48.912 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.902+0000 7fb2aad9d640 1 --2- 172.21.15.155:0/2799548861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2ac146f10 0x7fb2ac0aa9d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac143d80 tx=0x7fb2a400cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:48.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.902+0000 7fb2a2ffd640 10 client.?.objecter ms_handle_connect 0x7fb2ac146f10 2024-08-09T14:50:48.913 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.902+0000 7fb2a2ffd640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:48.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2a2ffd640 1 -- 172.21.15.155:0/2799548861 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4003b70 con 0x7fb2ac146f10 2024-08-09T14:50:48.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:48.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2a2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:48.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2a2ffd640 10 monclient: dump: 2024-08-09T14:50:48.914 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:48.914 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:48.914 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.914 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.914 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:48.915 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:48.915 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:48.915 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:48.915 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:48.915 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:48.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2a2ffd640 10 monclient: _finish_auth 0 2024-08-09T14:50:48.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2a2ffd640 10 monclient: _check_auth_tickets 2024-08-09T14:50:48.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:48.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2a2ffd640 1 -- 172.21.15.155:0/2799548861 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2a4003d10 con 0x7fb2ac146f10 2024-08-09T14:50:48.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2a2ffd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:48.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2a2ffd640 1 -- 172.21.15.155:0/2799548861 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a4005310 con 0x7fb2ac146f10 2024-08-09T14:50:48.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2b0ce9640 5 monclient: authenticate success, global_id 4255 2024-08-09T14:50:48.915 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:48.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:48.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2a2ffd640 10 monclient: dump: 2024-08-09T14:50:48.916 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:48.916 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:48.916 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.916 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:48.916 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:48.916 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:48.916 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:48.916 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:48.916 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:48.916 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:48.916 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2b0ce9640 10 monclient: _renew_subs 2024-08-09T14:50:48.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2b0ce9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:48.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/2799548861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2ac0abe50 con 0x7fb2ac146f10 2024-08-09T14:50:48.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2a959a640 4 set_mon_vals no callback set 2024-08-09T14:50:48.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2b0ce9640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:48.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2b0ce9640 10 monclient: _renew_subs 2024-08-09T14:50:48.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2b0ce9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:48.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/2799548861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2ac0aafc0 con 0x7fb2ac146f10 2024-08-09T14:50:48.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.906+0000 7fb2b0ce9640 1 librados: init done 2024-08-09T14:50:48.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.907+0000 7fb2b0ce9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:48.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.907+0000 7fb2b0ce9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:48.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.907+0000 7fb2b0ce9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:48.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.907+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/2799548861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2ac0b0540 con 0x7fb2ac146f10 2024-08-09T14:50:48.926 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.925+0000 7fb2a2ffd640 1 -- 172.21.15.155:0/2799548861 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb2a40054b0 con 0x7fb2ac146f10 2024-08-09T14:50:48.927 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.925+0000 7fb2a2ffd640 1 --2- 172.21.15.155:0/2799548861 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fb28c06d810 0x7fb28c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:48.927 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.925+0000 7fb2aa59c640 1 --2- 172.21.15.155:0/2799548861 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fb28c06d810 0x7fb28c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:48.927 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.925+0000 7fb2aa59c640 10 monclient: get_auth_request con 0x7fb28c06d810 auth_method 0 2024-08-09T14:50:48.927 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.926+0000 7fb2aa59c640 1 --2- 172.21.15.155:0/2799548861 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fb28c06d810 0x7fb28c06fcd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb29c02b8c0 tx=0x7fb29c03d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:48.927 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.926+0000 7fb2a2ffd640 10 client.4255.objecter ms_handle_connect 0x7fb28c06d810 2024-08-09T14:50:48.929 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.928+0000 7fb2a2ffd640 1 -- 172.21.15.155:0/2799548861 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7fb2a40057a0 con 0x7fb2ac146f10 2024-08-09T14:50:48.929 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.928+0000 7fb2a2ffd640 10 client.4255.objecter ms_dispatch 0x7fb2ac000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:48.929 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.928+0000 7fb2a2ffd640 3 client.4255.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:48.929 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.928+0000 7fb2a2ffd640 3 client.4255.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:48.929 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:48.928+0000 7fb2a2ffd640 20 client.4255.objecter dump_active .. 0 homeless 2024-08-09T14:50:49.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.171+0000 7fb2a2ffd640 1 -- 172.21.15.155:0/2799548861 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7fb2a40598c0 con 0x7fb2ac146f10 2024-08-09T14:50:49.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.171+0000 7fb2a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:49.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.171+0000 7fb2a2ffd640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:49.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.355+0000 7fb2b0ce9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:49.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.355+0000 7fb2b0ce9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:49.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.355+0000 7fb2b0ce9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:49.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.355+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/2799548861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb2ac0b3180 con 0x7fb2ac146f10 2024-08-09T14:50:49.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.392+0000 7fb2a2ffd640 1 -- 172.21.15.155:0/2799548861 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+14873 (secure 0 0 0) 0x7fb2a405dbc0 con 0x7fb2ac146f10 2024-08-09T14:50:49.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.392+0000 7fb2a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:49.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.392+0000 7fb2a2ffd640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:49.393 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:50:49.393 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":27,"fsid":"f3d21f4d-627f-4083-bf15-ae5f3fe632de","created":"2024-08-09T14:49:41.321598+0000","modified":"2024-08-09T14:50:47.578050+0000","last_up_change":"2024-08-09T14:50:39.237260+0000","last_in_change":"2024-08-09T14:49:46.978521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-09T14:50:30.139923+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-09T14:50:42.897895+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":27,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6526d474-661b-48e6-bcc7-6288c7477f78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":729861593},{"type":"v1","addr":"172.21.15.155:6822","nonce":729861593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":729861593},{"type":"v1","addr":"172.21.15.155:6828","nonce":729861593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":729861593},{"type":"v1","addr":"172.21.15.155:6833","nonce":729861593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":729861593},{"type":"v1","addr":"172.21.15.155:6831","nonce":729861593}]},"public_addr":"172.21.15.155:6822/729861593","cluster_addr":"172.21.15.155:6828/729861593","heartbeat_back_addr":"172.21.15.155:6833/729861593","heartbeat_front_addr":"172.21.15.155:6831/729861593","state":["exists","up"]},{"osd":1,"uuid":"87673c5a-b124-419e-98da-17a7b7fbdf25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6819","nonce":3129678288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6821","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6823","nonce":3129678288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6827","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6829","nonce":3129678288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6825","nonce":3129678288}]},"public_addr":"172.21.15.155:6819/3129678288","cluster_addr":"172.21.15.155:6823/3129678288","heartbeat_back_addr":"172.21.15.155:6829/3129678288","heartbeat_front_addr":"172.21.15.155:6825/3129678288","state":["exists","up"]},{"osd":2,"uuid":"a14a708a-903f-4de2-8228-79e79ed135e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":961143006},{"type":"v1","addr":"172.21.15.155:6803","nonce":961143006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":961143006},{"type":"v1","addr":"172.21.15.155:6805","nonce":961143006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":961143006},{"type":"v1","addr":"172.21.15.155:6809","nonce":961143006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":961143006},{"type":"v1","addr":"172.21.15.155:6807","nonce":961143006}]},"public_addr":"172.21.15.155:6803/961143006","cluster_addr":"172.21.15.155:6805/961143006","heartbeat_back_addr":"172.21.15.155:6809/961143006","heartbeat_front_addr":"172.21.15.155:6807/961143006","state":["exists","up"]},{"osd":3,"uuid":"8e139d50-e25c-4428-9edf-c696189cb40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6811","nonce":3967029756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6813","nonce":3967029756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6817","nonce":3967029756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6815","nonce":3967029756}]},"public_addr":"172.21.15.155:6811/3967029756","cluster_addr":"172.21.15.155:6813/3967029756","heartbeat_back_addr":"172.21.15.155:6817/3967029756","heartbeat_front_addr":"172.21.15.155:6815/3967029756","state":["exists","up"]},{"osd":4,"uuid":"d0331386-2f50-4bf9-b72e-24c80f9fe9a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6801","nonce":2229949751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6803","nonce":2229949751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6807","nonce":2229949751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6805","nonce":2229949751}]},"public_addr":"172.21.15.194:6801/2229949751","cluster_addr":"172.21.15.194:6803/2229949751","heartbeat_back_addr":"172.21.15.194:6807/2229949751","heartbeat_front_addr":"172.21.15.194:6805/2229949751","state":["exists","up"]},{"osd":5,"uuid":"0f901e55-2ba8-4144-9802-3a109884254c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6817","nonce":1013364343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6819","nonce":1013364343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6825","nonce":1013364343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6821","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6822","nonce":1013364343}]},"public_addr":"172.21.15.194:6817/1013364343","cluster_addr":"172.21.15.194:6819/1013364343","heartbeat_back_addr":"172.21.15.194:6825/1013364343","heartbeat_front_addr":"172.21.15.194:6822/1013364343","state":["exists","up"]},{"osd":6,"uuid":"fd4b8215-814e-4b5a-9877-0ab63818b0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6809","nonce":3077539270}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6811","nonce":3077539270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6815","nonce":3077539270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6813","nonce":3077539270}]},"public_addr":"172.21.15.194:6809/3077539270","cluster_addr":"172.21.15.194:6811/3077539270","heartbeat_back_addr":"172.21.15.194:6815/3077539270","heartbeat_front_addr":"172.21.15.194:6813/3077539270","state":["exists","up"]},{"osd":7,"uuid":"26ed84b6-639d-4ccd-bb1d-c64b593ada7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2670313675},{"type":"v1","addr":"172.21.15.194:6823","nonce":2670313675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2670313675},{"type":"v1","addr":"172.21.15.194:6827","nonce":2670313675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2670313675},{"type":"v1","addr":"172.21.15.194:6831","nonce":2670313675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2670313675},{"type":"v1","addr":"172.21.15.194:6829","nonce":2670313675}]},"public_addr":"172.21.15.194:6823/2670313675","cluster_addr":"172.21.15.194:6827/2670313675","heartbeat_back_addr":"172.21.15.194:6831/2670313675","heartbeat_front_addr":"172.21.15.194:6829/2670313675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:26.771476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:26.207562+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.576525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.163852+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.187804+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:26.108803+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.199884+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.757809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-09T14:50:49.395 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 10 librados: watch_flush enter 2024-08-09T14:50:49.395 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 10 librados: watch_flush exit 2024-08-09T14:50:49.395 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 20 client.4255.objecter shutdown clearing up homeless session... 2024-08-09T14:50:49.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 10 client.4255.objecter successfully canceled tick 2024-08-09T14:50:49.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/2799548861 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fb28c06d810 msgr2=0x7fb28c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:49.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 1 --2- 172.21.15.155:0/2799548861 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fb28c06d810 0x7fb28c06fcd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb29c02b8c0 tx=0x7fb29c03d040 comp rx=0 tx=0).stop 2024-08-09T14:50:49.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 10 monclient: shutdown 2024-08-09T14:50:49.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:49.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/2799548861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2ac146f10 msgr2=0x7fb2ac0aa9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:49.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 1 --2- 172.21.15.155:0/2799548861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2ac146f10 0x7fb2ac0aa9d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac143d80 tx=0x7fb2a400cb90 comp rx=0 tx=0).stop 2024-08-09T14:50:49.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/2799548861 shutdown_connections 2024-08-09T14:50:49.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 1 --2- 172.21.15.155:0/2799548861 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fb28c06d810 0x7fb28c06fcd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:49.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 1 --2- 172.21.15.155:0/2799548861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2ac146f10 0x7fb2ac0aa9d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:49.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 1 --2- 172.21.15.155:0/2799548861 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb2ac0b22e0 0x7fb2ac0aa490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:49.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 1 --2- 172.21.15.155:0/2799548861 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb2ac0b0160 0x7fb2ac14c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:49.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/2799548861 >> 172.21.15.155:0/2799548861 conn(0x7fb2ac06dbd0 msgr2=0x7fb2ac0b53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:49.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/2799548861 shutdown_connections 2024-08-09T14:50:49.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 1 -- 172.21.15.155:0/2799548861 wait complete. 2024-08-09T14:50:49.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.393+0000 7fb2b0ce9640 1 librados: shutdown 2024-08-09T14:50:49.406 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-08-09T14:50:49.406 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-08-09T14:50:49.505 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.504+0000 7f56c81d2640 10 monclient: get_monmap_and_config 2024-08-09T14:50:49.506 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.504+0000 7f56c81d2640 10 monclient: build_initial_monmap 2024-08-09T14:50:49.506 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.504+0000 7f56c81d2640 10 monclient: monmap: 2024-08-09T14:50:49.506 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:49.506 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:49.506 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:49.505702+0000 2024-08-09T14:50:49.506 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:49.505702+0000 2024-08-09T14:50:49.506 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:49.506 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:49.506 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:49.506 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:49.507 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:49.507 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:49.507 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:49.507 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:49.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.505+0000 7f56c81d2640 1 Processor -- start 2024-08-09T14:50:49.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.505+0000 7f56c81d2640 1 -- start start 2024-08-09T14:50:49.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.505+0000 7f56c81d2640 10 monclient: init 2024-08-09T14:50:49.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c81d2640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:49.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c81d2640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-08-09T14:50:49.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c81d2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f56c00af230 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:49.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c81d2640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c00733b0 0x7f56c00ae6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:49.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c81d2640 10 monclient(hunting): picked mon.noname-b con 0x7f56c00733b0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:49.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c81d2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56c013cb20 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:49.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c81d2640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56c00ac5c0 con 0x7f56c013cb20 2024-08-09T14:50:49.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c81d2640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56c00aec30 con 0x7f56c00af230 2024-08-09T14:50:49.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c81d2640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:49.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c81d2640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f56c00ae280 con 0x7f56c00733b0 2024-08-09T14:50:49.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c81d2640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:49.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c81d2640 10 monclient(hunting): authenticate will time out at 1256.543165s 2024-08-09T14:50:49.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c576d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c00733b0 0x7f56c00ae6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:49.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c676f640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f56c013cb20 0x7f56c013cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:34490/0 (socket says 172.21.15.155:34490) 2024-08-09T14:50:49.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c676f640 1 -- 172.21.15.155:0/334170949 learned_addr learned my addr 172.21.15.155:0/334170949 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:49.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c576d640 10 monclient(hunting): get_auth_request con 0x7f56c00733b0 auth_method 0 2024-08-09T14:50:49.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c576d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:49.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c576d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:49.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.506+0000 7f56c576d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:49.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.507+0000 7f56c576d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:49.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.507+0000 7f56c576d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:49.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.507+0000 7f56c576d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.509+0000 7f56c576d640 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2024-08-09T14:50:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.509+0000 7f56c576d640 1 -- 172.21.15.155:0/334170949 >> v1:172.21.15.194:6789/0 conn(0x7f56c00af230 legacy=0x7f56c00ab6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.509+0000 7f56c576d640 1 -- 172.21.15.155:0/334170949 >> v1:172.21.15.155:6789/0 conn(0x7f56c013cb20 legacy=0x7f56c013cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.509+0000 7f56c576d640 10 monclient: _finish_hunting 0 2024-08-09T14:50:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.509+0000 7f56c576d640 1 monclient: found mon.noname-b 2024-08-09T14:50:49.512 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.509+0000 7f56c576d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.509+0000 7f56c576d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.509+0000 7f56c576d640 1 -- 172.21.15.155:0/334170949 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c00aca10 con 0x7f56c00733b0 2024-08-09T14:50:49.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.511+0000 7f56c576d640 1 --2- 172.21.15.155:0/334170949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c00733b0 0x7f56c00ae6f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f56b00097d0 tx=0x7f56b002dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f3cf15e97237f49 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:49.516 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c4f6c640 1 -- 172.21.15.155:0/334170949 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b002f870 con 0x7f56c00733b0 2024-08-09T14:50:49.516 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c4f6c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:49.516 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c4f6c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:49.517 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c4f6c640 10 monclient: dump: 2024-08-09T14:50:49.517 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:49.517 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:49.517 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:49.517 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:49.517 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:49.517 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:49.517 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:49.517 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:49.517 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:49.517 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:49.517 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c4f6c640 10 monclient: _finish_auth 0 2024-08-09T14:50:49.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c4f6c640 10 monclient: _check_auth_tickets 2024-08-09T14:50:49.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c4f6c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:49.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c4f6c640 1 -- 172.21.15.155:0/334170949 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56b002fa10 con 0x7f56c00733b0 2024-08-09T14:50:49.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c4f6c640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:49.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c4f6c640 1 -- 172.21.15.155:0/334170949 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b002fde0 con 0x7f56c00733b0 2024-08-09T14:50:49.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c81d2640 5 monclient: authenticate success, global_id 4256 2024-08-09T14:50:49.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c81d2640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:49.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c81d2640 4 set_mon_vals no callback set 2024-08-09T14:50:49.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c4f6c640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:49.518 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c4f6c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:49.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.515+0000 7f56c4f6c640 10 monclient: dump: 2024-08-09T14:50:49.519 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:49.519 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:49.519 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:49.519 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:49.519 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:49.519 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:49.519 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:49.520 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:49.520 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:49.520 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:49.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.516+0000 7f56c81d2640 10 monclient: shutdown 2024-08-09T14:50:49.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.516+0000 7f56c81d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:49.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.516+0000 7f56c81d2640 1 -- 172.21.15.155:0/334170949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c00733b0 msgr2=0x7f56c00ae6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:49.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.516+0000 7f56c81d2640 1 --2- 172.21.15.155:0/334170949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c00733b0 0x7f56c00ae6f0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f56b00097d0 tx=0x7f56b002dab0 comp rx=0 tx=0).stop 2024-08-09T14:50:49.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.516+0000 7f56c81d2640 1 -- 172.21.15.155:0/334170949 shutdown_connections 2024-08-09T14:50:49.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.516+0000 7f56c81d2640 1 --2- 172.21.15.155:0/334170949 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c00733b0 0x7f56c00ae6f0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:49.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.516+0000 7f56c81d2640 1 -- 172.21.15.155:0/334170949 >> 172.21.15.155:0/334170949 conn(0x7f56c00b45c0 msgr2=0x7f56c00b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:49.523 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.516+0000 7f56c81d2640 1 -- 172.21.15.155:0/334170949 shutdown_connections 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.516+0000 7f56c81d2640 1 -- 172.21.15.155:0/334170949 wait complete. 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.516+0000 7f56c81d2640 10 monclient: build_initial_monmap 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.516+0000 7f56c81d2640 10 monclient: monmap: 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 1 librados: starting msgr at 2024-08-09T14:50:49.524 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 1 librados: starting objecter 2024-08-09T14:50:49.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 1 Processor -- start 2024-08-09T14:50:49.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 1 -- start start 2024-08-09T14:50:49.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 1 librados: setting wanted keys 2024-08-09T14:50:49.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 1 librados: calling monclient init 2024-08-09T14:50:49.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 10 monclient: init 2024-08-09T14:50:49.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:49.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 10 monclient: _add_conns ranks=[0,2,1] 2024-08-09T14:50:49.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56c00733b0 0x7f56c01400b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:49.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 10 monclient(hunting): picked mon.noname-a con 0x7f56c00733b0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:49.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c00af230 0x7f56c0142a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:49.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 10 monclient(hunting): picked mon.noname-c con 0x7f56c00af230 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:49.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56c013cb20 0x7f56c0142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:49.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 10 monclient(hunting): picked mon.noname-b con 0x7f56c013cb20 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:49.525 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:49.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f56c00af850 con 0x7f56c00733b0 2024-08-09T14:50:49.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:49.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f56c0072c60 con 0x7f56c013cb20 2024-08-09T14:50:49.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:49.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f56c00af9d0 con 0x7f56c00af230 2024-08-09T14:50:49.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:49.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.517+0000 7f56c81d2640 10 monclient(hunting): authenticate will time out at 1256.554922s 2024-08-09T14:50:49.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c576d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c00af230 0x7f56c0142a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:49.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c5f6e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56c00733b0 0x7f56c01400b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:49.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c676f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56c013cb20 0x7f56c0142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:49.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c576d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c00af230 0x7f56c0142a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54598/0 (socket says 172.21.15.155:54598) 2024-08-09T14:50:49.526 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c576d640 1 -- 172.21.15.155:0/1738041005 learned_addr learned my addr 172.21.15.155:0/1738041005 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:49.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c576d640 10 monclient(hunting): get_auth_request con 0x7f56c00af230 auth_method 0 2024-08-09T14:50:49.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c576d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:49.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c576d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:49.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c576d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:49.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c676f640 10 monclient(hunting): get_auth_request con 0x7f56c013cb20 auth_method 0 2024-08-09T14:50:49.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c676f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:49.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c676f640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:49.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c676f640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:49.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c5f6e640 10 monclient(hunting): get_auth_request con 0x7f56c00733b0 auth_method 0 2024-08-09T14:50:49.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c5f6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:49.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c5f6e640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:49.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.518+0000 7f56c5f6e640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:49.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.519+0000 7f56c576d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:49.527 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.519+0000 7f56c576d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:49.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.519+0000 7f56c576d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:49.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.519+0000 7f56c676f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:49.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.519+0000 7f56c676f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:49.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.519+0000 7f56c676f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:49.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.519+0000 7f56c5f6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:49.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.519+0000 7f56c5f6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:49.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.519+0000 7f56c5f6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:49.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.522+0000 7f56c676f640 10 monclient(hunting): handle_auth_done global_id 4261 payload 824 2024-08-09T14:50:49.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.522+0000 7f56c676f640 1 -- 172.21.15.155:0/1738041005 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c00af230 msgr2=0x7f56c0142a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:49.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.522+0000 7f56c676f640 1 --2- 172.21.15.155:0/1738041005 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c00af230 0x7f56c0142a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:49.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.522+0000 7f56c676f640 1 -- 172.21.15.155:0/1738041005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56c00733b0 msgr2=0x7f56c01400b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:49.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.522+0000 7f56c676f640 1 --2- 172.21.15.155:0/1738041005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56c00733b0 0x7f56c01400b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:49.528 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.522+0000 7f56c676f640 10 monclient: _finish_hunting 0 2024-08-09T14:50:49.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.522+0000 7f56c676f640 1 monclient: found mon.noname-b 2024-08-09T14:50:49.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.522+0000 7f56c676f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:49.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.522+0000 7f56c676f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:49.529 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.522+0000 7f56c676f640 1 -- 172.21.15.155:0/1738041005 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56c006f320 con 0x7f56c013cb20 2024-08-09T14:50:49.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.524+0000 7f56c676f640 1 --2- 172.21.15.155:0/1738041005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56c013cb20 0x7f56c0142fc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f56b800cad0 tx=0x7f56b800cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:49.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.524+0000 7f56b67fc640 10 client.?.objecter ms_handle_connect 0x7f56c013cb20 2024-08-09T14:50:49.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.524+0000 7f56b67fc640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:49.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56b67fc640 1 -- 172.21.15.155:0/1738041005 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8017070 con 0x7f56c013cb20 2024-08-09T14:50:49.530 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:49.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56b67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:49.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56b67fc640 10 monclient: dump: 2024-08-09T14:50:49.531 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:49.531 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:49.531 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:49.531 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:49.531 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:49.531 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:49.533 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:49.533 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:49.533 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:49.533 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:49.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56b67fc640 10 monclient: _finish_auth 0 2024-08-09T14:50:49.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56b67fc640 10 monclient: _check_auth_tickets 2024-08-09T14:50:49.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:49.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56b67fc640 1 -- 172.21.15.155:0/1738041005 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f56b800ab70 con 0x7f56c013cb20 2024-08-09T14:50:49.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56b67fc640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:49.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56b67fc640 1 -- 172.21.15.155:0/1738041005 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56b8012400 con 0x7f56c013cb20 2024-08-09T14:50:49.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:49.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56b67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:49.534 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56b67fc640 10 monclient: dump: 2024-08-09T14:50:49.534 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:49.535 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:49.535 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:49.535 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:49.535 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:49.535 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:49.535 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:49.535 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:49.535 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:49.535 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:49.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56c81d2640 5 monclient: authenticate success, global_id 4261 2024-08-09T14:50:49.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56c81d2640 10 monclient: _renew_subs 2024-08-09T14:50:49.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56c81d2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:49.535 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56c81d2640 1 -- 172.21.15.155:0/1738041005 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56c006f600 con 0x7f56c013cb20 2024-08-09T14:50:49.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.527+0000 7f56b7fff640 4 set_mon_vals no callback set 2024-08-09T14:50:49.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.528+0000 7f56c81d2640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:49.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.528+0000 7f56c81d2640 10 monclient: _renew_subs 2024-08-09T14:50:49.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.528+0000 7f56c81d2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:49.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.528+0000 7f56c81d2640 1 -- 172.21.15.155:0/1738041005 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56c0143520 con 0x7f56c013cb20 2024-08-09T14:50:49.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.528+0000 7f56c81d2640 1 librados: init done 2024-08-09T14:50:49.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.528+0000 7f56c81d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:49.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.528+0000 7f56c81d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:49.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.528+0000 7f56c81d2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:49.536 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.528+0000 7f56c81d2640 1 -- 172.21.15.155:0/1738041005 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56c00ae6f0 con 0x7f56c013cb20 2024-08-09T14:50:49.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.547+0000 7f56b67fc640 1 -- 172.21.15.155:0/1738041005 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f56b801a070 con 0x7f56c013cb20 2024-08-09T14:50:49.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.548+0000 7f56b67fc640 1 --2- 172.21.15.155:0/1738041005 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f56a406d7c0 0x7f56a406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:49.549 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.548+0000 7f56c5f6e640 1 --2- 172.21.15.155:0/1738041005 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f56a406d7c0 0x7f56a406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:49.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.548+0000 7f56c5f6e640 10 monclient: get_auth_request con 0x7f56a406d7c0 auth_method 0 2024-08-09T14:50:49.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.548+0000 7f56b67fc640 10 client.4261.objecter ms_handle_connect 0x7f56a406d7c0 2024-08-09T14:50:49.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.548+0000 7f56c5f6e640 1 --2- 172.21.15.155:0/1738041005 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f56a406d7c0 0x7f56a406fc80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f56bc000f60 tx=0x7f56bc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:49.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.552+0000 7f56b67fc640 1 -- 172.21.15.155:0/1738041005 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f56b80575f0 con 0x7f56c013cb20 2024-08-09T14:50:49.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.552+0000 7f56b67fc640 10 client.4261.objecter ms_dispatch 0x7f56c0000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:49.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.552+0000 7f56b67fc640 3 client.4261.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:49.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.552+0000 7f56b67fc640 3 client.4261.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:49.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.552+0000 7f56b67fc640 20 client.4261.objecter dump_active .. 0 homeless 2024-08-09T14:50:49.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.796+0000 7f56b67fc640 1 -- 172.21.15.155:0/1738041005 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f56b8059550 con 0x7f56c013cb20 2024-08-09T14:50:49.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.796+0000 7f56b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:49.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:49.796+0000 7f56b67fc640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:50.001 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.000+0000 7f56c81d2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:50.001 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.000+0000 7f56c81d2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:50.001 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.000+0000 7f56c81d2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.001 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.000+0000 7f56c81d2640 1 -- 172.21.15.155:0/1738041005 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f56c00b3660 con 0x7f56c013cb20 2024-08-09T14:50:50.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.028+0000 7f56b67fc640 1 -- 172.21.15.155:0/1738041005 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) ==== 74+0+14873 (secure 0 0 0) 0x7f56b805d850 con 0x7f56c013cb20 2024-08-09T14:50:50.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.028+0000 7f56b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-08-09T14:50:50.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.028+0000 7f56b67fc640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:50.029 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:50:50.029 INFO:teuthology.orchestra.run.smithi155.stdout:{"epoch":27,"fsid":"f3d21f4d-627f-4083-bf15-ae5f3fe632de","created":"2024-08-09T14:49:41.321598+0000","modified":"2024-08-09T14:50:47.578050+0000","last_up_change":"2024-08-09T14:50:39.237260+0000","last_in_change":"2024-08-09T14:49:46.978521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-09T14:50:30.139923+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-08-09T14:50:42.897895+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":27,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"6526d474-661b-48e6-bcc7-6288c7477f78","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6820","nonce":729861593},{"type":"v1","addr":"172.21.15.155:6822","nonce":729861593}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6826","nonce":729861593},{"type":"v1","addr":"172.21.15.155:6828","nonce":729861593}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6832","nonce":729861593},{"type":"v1","addr":"172.21.15.155:6833","nonce":729861593}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6830","nonce":729861593},{"type":"v1","addr":"172.21.15.155:6831","nonce":729861593}]},"public_addr":"172.21.15.155:6822/729861593","cluster_addr":"172.21.15.155:6828/729861593","heartbeat_back_addr":"172.21.15.155:6833/729861593","heartbeat_front_addr":"172.21.15.155:6831/729861593","state":["exists","up"]},{"osd":1,"uuid":"87673c5a-b124-419e-98da-17a7b7fbdf25","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6818","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6819","nonce":3129678288}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6821","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6823","nonce":3129678288}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6827","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6829","nonce":3129678288}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6824","nonce":3129678288},{"type":"v1","addr":"172.21.15.155:6825","nonce":3129678288}]},"public_addr":"172.21.15.155:6819/3129678288","cluster_addr":"172.21.15.155:6823/3129678288","heartbeat_back_addr":"172.21.15.155:6829/3129678288","heartbeat_front_addr":"172.21.15.155:6825/3129678288","state":["exists","up"]},{"osd":2,"uuid":"a14a708a-903f-4de2-8228-79e79ed135e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6802","nonce":961143006},{"type":"v1","addr":"172.21.15.155:6803","nonce":961143006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6804","nonce":961143006},{"type":"v1","addr":"172.21.15.155:6805","nonce":961143006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6808","nonce":961143006},{"type":"v1","addr":"172.21.15.155:6809","nonce":961143006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6806","nonce":961143006},{"type":"v1","addr":"172.21.15.155:6807","nonce":961143006}]},"public_addr":"172.21.15.155:6803/961143006","cluster_addr":"172.21.15.155:6805/961143006","heartbeat_back_addr":"172.21.15.155:6809/961143006","heartbeat_front_addr":"172.21.15.155:6807/961143006","state":["exists","up"]},{"osd":3,"uuid":"8e139d50-e25c-4428-9edf-c696189cb40e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6810","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6811","nonce":3967029756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6812","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6813","nonce":3967029756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6816","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6817","nonce":3967029756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.155:6814","nonce":3967029756},{"type":"v1","addr":"172.21.15.155:6815","nonce":3967029756}]},"public_addr":"172.21.15.155:6811/3967029756","cluster_addr":"172.21.15.155:6813/3967029756","heartbeat_back_addr":"172.21.15.155:6817/3967029756","heartbeat_front_addr":"172.21.15.155:6815/3967029756","state":["exists","up"]},{"osd":4,"uuid":"d0331386-2f50-4bf9-b72e-24c80f9fe9a8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6801","nonce":2229949751}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6803","nonce":2229949751}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6807","nonce":2229949751}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2229949751},{"type":"v1","addr":"172.21.15.194:6805","nonce":2229949751}]},"public_addr":"172.21.15.194:6801/2229949751","cluster_addr":"172.21.15.194:6803/2229949751","heartbeat_back_addr":"172.21.15.194:6807/2229949751","heartbeat_front_addr":"172.21.15.194:6805/2229949751","state":["exists","up"]},{"osd":5,"uuid":"0f901e55-2ba8-4144-9802-3a109884254c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6817","nonce":1013364343}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6819","nonce":1013364343}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6825","nonce":1013364343}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6821","nonce":1013364343},{"type":"v1","addr":"172.21.15.194:6822","nonce":1013364343}]},"public_addr":"172.21.15.194:6817/1013364343","cluster_addr":"172.21.15.194:6819/1013364343","heartbeat_back_addr":"172.21.15.194:6825/1013364343","heartbeat_front_addr":"172.21.15.194:6822/1013364343","state":["exists","up"]},{"osd":6,"uuid":"fd4b8215-814e-4b5a-9877-0ab63818b0ba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6809","nonce":3077539270}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6811","nonce":3077539270}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6815","nonce":3077539270}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":3077539270},{"type":"v1","addr":"172.21.15.194:6813","nonce":3077539270}]},"public_addr":"172.21.15.194:6809/3077539270","cluster_addr":"172.21.15.194:6811/3077539270","heartbeat_back_addr":"172.21.15.194:6815/3077539270","heartbeat_front_addr":"172.21.15.194:6813/3077539270","state":["exists","up"]},{"osd":7,"uuid":"26ed84b6-639d-4ccd-bb1d-c64b593ada7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":2670313675},{"type":"v1","addr":"172.21.15.194:6823","nonce":2670313675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":2670313675},{"type":"v1","addr":"172.21.15.194:6827","nonce":2670313675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":2670313675},{"type":"v1","addr":"172.21.15.194:6831","nonce":2670313675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":2670313675},{"type":"v1","addr":"172.21.15.194:6829","nonce":2670313675}]},"public_addr":"172.21.15.194:6823/2670313675","cluster_addr":"172.21.15.194:6827/2670313675","heartbeat_back_addr":"172.21.15.194:6831/2670313675","heartbeat_front_addr":"172.21.15.194:6829/2670313675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:26.771476+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:26.207562+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.576525+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.163852+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.187804+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:26.108803+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.199884+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-09T14:50:25.757809+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-09T14:50:50.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 10 librados: watch_flush enter 2024-08-09T14:50:50.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 10 librados: watch_flush exit 2024-08-09T14:50:50.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 20 client.4261.objecter shutdown clearing up homeless session... 2024-08-09T14:50:50.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 10 client.4261.objecter successfully canceled tick 2024-08-09T14:50:50.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 1 -- 172.21.15.155:0/1738041005 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f56a406d7c0 msgr2=0x7f56a406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 1 --2- 172.21.15.155:0/1738041005 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f56a406d7c0 0x7f56a406fc80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f56bc000f60 tx=0x7f56bc009040 comp rx=0 tx=0).stop 2024-08-09T14:50:50.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 10 monclient: shutdown 2024-08-09T14:50:50.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 1 -- 172.21.15.155:0/1738041005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56c013cb20 msgr2=0x7f56c0142fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 1 --2- 172.21.15.155:0/1738041005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56c013cb20 0x7f56c0142fc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f56b800cad0 tx=0x7f56b800cf90 comp rx=0 tx=0).stop 2024-08-09T14:50:50.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 1 -- 172.21.15.155:0/1738041005 shutdown_connections 2024-08-09T14:50:50.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 1 --2- 172.21.15.155:0/1738041005 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f56a406d7c0 0x7f56a406fc80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 1 --2- 172.21.15.155:0/1738041005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56c013cb20 0x7f56c0142fc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 1 --2- 172.21.15.155:0/1738041005 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f56c00af230 0x7f56c0142a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 1 --2- 172.21.15.155:0/1738041005 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f56c00733b0 0x7f56c01400b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 1 -- 172.21.15.155:0/1738041005 >> 172.21.15.155:0/1738041005 conn(0x7f56c00b45c0 msgr2=0x7f56c013a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 1 -- 172.21.15.155:0/1738041005 shutdown_connections 2024-08-09T14:50:50.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 1 -- 172.21.15.155:0/1738041005 wait complete. 2024-08-09T14:50:50.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.029+0000 7f56c81d2640 1 librados: shutdown 2024-08-09T14:50:50.043 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-08-09T14:50:50.043 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-08-09T14:50:50.043 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-08-09T14:50:50.044 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-08-09T14:50:50.044 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-08-09T14:50:50.045 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-08-09T14:50:50.045 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-08-09T14:50:50.045 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-08-09T14:50:50.212 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.211+0000 7fac63e17640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.212 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.211+0000 7fac63e17640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.213 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.211+0000 7fac63e17640 10 monclient: monmap: 2024-08-09T14:50:50.213 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.213 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.213 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.212319+0000 2024-08-09T14:50:50.213 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.212319+0000 2024-08-09T14:50:50.213 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.213 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.213 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.213 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.213 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.213 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.213 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.213 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.215+0000 7fac63e17640 1 Processor -- start 2024-08-09T14:50:50.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.215+0000 7fac63e17640 1 -- start start 2024-08-09T14:50:50.216 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.215+0000 7fac63e17640 10 monclient: init 2024-08-09T14:50:50.217 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.216+0000 7fa69cbed640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.217 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.216+0000 7fa69cbed640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.217 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.216+0000 7fa69cbed640 10 monclient: monmap: 2024-08-09T14:50:50.217 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.217 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.217 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.217233+0000 2024-08-09T14:50:50.217 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.217233+0000 2024-08-09T14:50:50.217 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.217 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.218 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.218 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.218 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.218 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.218 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.218 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.218 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.215+0000 7f3d22c22640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.219 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.216+0000 7fac63e17640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.219 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.215+0000 7f3d22c22640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.219 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.215+0000 7f3d22c22640 10 monclient: monmap: 2024-08-09T14:50:50.219 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.219 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.219 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.216448+0000 2024-08-09T14:50:50.219 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.216448+0000 2024-08-09T14:50:50.219 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.219 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.219 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.220 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.220 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.220 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.220 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.220 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.220 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.216+0000 7fac63e17640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-08-09T14:50:50.220 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.216+0000 7fac63e17640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac5c0b0160 0x7fac5c0b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.216+0000 7fac63e17640 10 monclient(hunting): picked mon.noname-b con 0x7fac5c0b0160 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f3d22c22640 1 Processor -- start 2024-08-09T14:50:50.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7fac63e17640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac5c0b22e0 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:50.221 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7fa69cbed640 1 Processor -- start 2024-08-09T14:50:50.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7fac63e17640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fac5c0b26e0 0x7fac5c0b3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7fac63e17640 10 monclient(hunting): picked mon.noname-c con 0x7fac5c0b26e0 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:50.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.215+0000 7f2484731640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.215+0000 7f2484731640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.222 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.215+0000 7f2484731640 10 monclient: monmap: 2024-08-09T14:50:50.222 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.223 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.223 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.216440+0000 2024-08-09T14:50:50.223 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.216440+0000 2024-08-09T14:50:50.223 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.223 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.223 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.223 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.223 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.223 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.223 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.223 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.216+0000 7f2484731640 1 Processor -- start 2024-08-09T14:50:50.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.216+0000 7f2484731640 1 -- start start 2024-08-09T14:50:50.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.216+0000 7f2484731640 10 monclient: init 2024-08-09T14:50:50.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7fac63e17640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac5c1470e0 con 0x7fac5c0b22e0 2024-08-09T14:50:50.224 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7fac63e17640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7fac63e17640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7fac5c1461c0 con 0x7fac5c0b26e0 2024-08-09T14:50:50.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7fac63e17640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7fac63e17640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fac5c0b3d80 con 0x7fac5c0b0160 2024-08-09T14:50:50.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7fac63e17640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7fac63e17640 10 monclient(hunting): authenticate will time out at 1257.254551s 2024-08-09T14:50:50.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f3d22c22640 1 -- start start 2024-08-09T14:50:50.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f3d22c22640 10 monclient: init 2024-08-09T14:50:50.225 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f2484731640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f2484731640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-08-09T14:50:50.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f2484731640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f247c0ab520 0x7f247c0aa5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f2484731640 10 monclient(hunting): picked mon.noname-c con 0x7f247c0ab520 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:50.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f2484731640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f247c0aefc0 0x7f247c1534e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f2484731640 10 monclient(hunting): picked mon.noname-b con 0x7f247c0aefc0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f2484731640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f247c0af400 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:50.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f2484731640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f247c0acd30 con 0x7f247c0af400 2024-08-09T14:50:50.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f2484731640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f2484731640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f247c0ae990 con 0x7f247c0ab520 2024-08-09T14:50:50.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f2484731640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f2484731640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f247c0aaaf0 con 0x7f247c0aefc0 2024-08-09T14:50:50.226 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f2484731640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7f2484731640 10 monclient(hunting): authenticate will time out at 1257.254773s 2024-08-09T14:50:50.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f3d22c22640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.227 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fac623b4640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fac5c0b26e0 0x7fac5c0b3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7fa69cbed640 1 -- start start 2024-08-09T14:50:50.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.217+0000 7fa69cbed640 10 monclient: init 2024-08-09T14:50:50.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fac61bb3640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac5c0b0160 0x7fac5c0b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.228 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fa69cbed640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fa69cbed640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-08-09T14:50:50.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fa69cbed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6980aa820 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:50.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fa69cbed640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa6980ae9c0 0x7fa698147030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fa69cbed640 10 monclient(hunting): picked mon.noname-a con 0x7fa6980ae9c0 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:50.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fa69cbed640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fa6981449a0 0x7fa698147570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fa69cbed640 10 monclient(hunting): picked mon.noname-c con 0x7fa6981449a0 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:50.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fa69cbed640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6980ace10 con 0x7fa6980aa820 2024-08-09T14:50:50.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fa69cbed640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fa69cbed640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7fa6980ab690 con 0x7fa6980ae9c0 2024-08-09T14:50:50.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fa69cbed640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fa69cbed640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7fa6980ab130 con 0x7fa6981449a0 2024-08-09T14:50:50.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fa69cbed640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.229 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fa69cbed640 10 monclient(hunting): authenticate will time out at 1257.255494s 2024-08-09T14:50:50.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fac61bb3640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac5c0b0160 0x7fac5c0b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54608/0 (socket says 172.21.15.155:54608) 2024-08-09T14:50:50.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fac61bb3640 1 -- 172.21.15.155:0/2895706432 learned_addr learned my addr 172.21.15.155:0/2895706432 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.230 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fac61bb3640 10 monclient(hunting): get_auth_request con 0x7fac5c0b0160 auth_method 0 2024-08-09T14:50:50.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f3d22c22640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-08-09T14:50:50.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f3d22c22640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d14006510 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:50.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f3d22c22640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3d140c34e0 0x7f3d140c7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f3d22c22640 10 monclient(hunting): picked mon.noname-a con 0x7f3d140c34e0 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:50.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f3d22c22640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d140c7640 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:50.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f3d22c22640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d1402c4f0 con 0x7f3d140c7640 2024-08-09T14:50:50.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f3d22c22640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d140c1020 con 0x7f3d14006510 2024-08-09T14:50:50.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f3d22c22640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f3d22c22640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f3d140c6c00 con 0x7f3d140c34e0 2024-08-09T14:50:50.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f3d22c22640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.231 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f3d22c22640 10 monclient(hunting): authenticate will time out at 1257.255346s 2024-08-09T14:50:50.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fac61bb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fac61bb3640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fac61bb3640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fac623b4640 10 monclient(hunting): get_auth_request con 0x7fac5c0b26e0 auth_method 0 2024-08-09T14:50:50.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fac623b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fac623b4640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7fac623b4640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f24824cd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f247c0ab520 0x7f247c0aa5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.232 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f2482cce640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f247c0af400 0x7f247c153a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:34498/0 (socket says 172.21.15.155:34498) 2024-08-09T14:50:50.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f2482cce640 1 -- 172.21.15.155:0/3320873738 learned_addr learned my addr 172.21.15.155:0/3320873738 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:50.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f6cae0a0640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f6cae0a0640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.233 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.218+0000 7f6cae0a0640 10 monclient: monmap: 2024-08-09T14:50:50.233 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.233 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.233 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.219970+0000 2024-08-09T14:50:50.233 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.219970+0000 2024-08-09T14:50:50.233 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.234 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.234 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.234 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.234 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.234 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.234 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.234 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.234 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.219+0000 7fac61bb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.234 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.219+0000 7fac61bb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.234 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.219+0000 7fac61bb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.235 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.219+0000 7f6cae0a0640 1 Processor -- start 2024-08-09T14:50:50.235 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f6cae0a0640 1 -- start start 2024-08-09T14:50:50.235 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f6cae0a0640 10 monclient: init 2024-08-09T14:50:50.235 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f3d2141f640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3d140c34e0 0x7f3d140c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f24824cd640 10 monclient(hunting): get_auth_request con 0x7f247c0ab520 auth_method 0 2024-08-09T14:50:50.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f24824cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f24824cd640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f24824cd640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f0bf78e1640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f0bf78e1640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f0bf78e1640 10 monclient: monmap: 2024-08-09T14:50:50.236 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.236 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.236 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.221642+0000 2024-08-09T14:50:50.237 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.221642+0000 2024-08-09T14:50:50.237 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.237 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.237 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.237 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.237 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.237 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.237 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.237 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.238 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f3d2141f640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f3d140c34e0 0x7f3d140c7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43430/0 (socket says 172.21.15.155:43430) 2024-08-09T14:50:50.238 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f3d2141f640 1 -- 172.21.15.155:0/3860018960 learned_addr learned my addr 172.21.15.155:0/3860018960 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.238 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f3d2141f640 10 monclient(hunting): get_auth_request con 0x7f3d140c34e0 auth_method 0 2024-08-09T14:50:50.238 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f3d2141f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.238 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f3d2141f640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.238 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f6cae0a0640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f6cae0a0640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-08-09T14:50:50.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f94bca23640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f94bca23640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f94bca23640 10 monclient: monmap: 2024-08-09T14:50:50.239 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.239 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.239 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.222346+0000 2024-08-09T14:50:50.239 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.222346+0000 2024-08-09T14:50:50.239 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.240 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.240 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.240 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.240 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.240 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.240 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.240 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f3d2141f640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f6cae0a0640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f6ca80aa820 0x7f6ca8144290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.220+0000 7f6cae0a0640 10 monclient(hunting): picked mon.noname-c con 0x7f6ca80aa820 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:50.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7fa695d9b640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa6980ae9c0 0x7fa698147030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f6cae0a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ca80ae9c0 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:50.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f6cae0a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ca80aac00 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:50.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f6cae0a0640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca80ace10 con 0x7f6ca80aac00 2024-08-09T14:50:50.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f6cae0a0640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca80ab130 con 0x7f6ca80ae9c0 2024-08-09T14:50:50.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f6cae0a0640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f6cae0a0640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f6ca80ab690 con 0x7f6ca80aa820 2024-08-09T14:50:50.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f6cae0a0640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f6cae0a0640 10 monclient(hunting): authenticate will time out at 1257.258066s 2024-08-09T14:50:50.243 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f0bf78e1640 1 Processor -- start 2024-08-09T14:50:50.243 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f0bf78e1640 1 -- start start 2024-08-09T14:50:50.243 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7fac623b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.243 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7fac623b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.243 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7fac623b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.243 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f24824cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7f24824cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7f24824cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.221+0000 7f0bf78e1640 10 monclient: init 2024-08-09T14:50:50.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7fa695d9b640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa6980ae9c0 0x7fa698147030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43434/0 (socket says 172.21.15.155:43434) 2024-08-09T14:50:50.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7fa695d9b640 1 -- 172.21.15.155:0/3741380805 learned_addr learned my addr 172.21.15.155:0/3741380805 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7fa695d9b640 10 monclient(hunting): get_auth_request con 0x7fa6980ae9c0 auth_method 0 2024-08-09T14:50:50.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7fa695d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7fa695d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7fa695d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7f94bca23640 1 Processor -- start 2024-08-09T14:50:50.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7f94bca23640 1 -- start start 2024-08-09T14:50:50.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7f94bca23640 10 monclient: init 2024-08-09T14:50:50.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7f0bf78e1640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7f0bf78e1640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-08-09T14:50:50.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7f0bf78e1640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0bf0071010 0x7f0bf013cb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7f0bf78e1640 10 monclient(hunting): picked mon.noname-a con 0x7f0bf0071010 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:50.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7f0bf78e1640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bf00afb50 0x7f0bf013d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7f0bf78e1640 10 monclient(hunting): picked mon.noname-b con 0x7f0bf00afb50 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.222+0000 7f0bf78e1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0bf00713f0 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:50.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f0bf78e1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bf0073e60 con 0x7f0bf00713f0 2024-08-09T14:50:50.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f0bf78e1640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f0bf78e1640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0bf013d5a0 con 0x7f0bf0071010 2024-08-09T14:50:50.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f0bf78e1640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f0bf78e1640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf0070860 con 0x7f0bf00afb50 2024-08-09T14:50:50.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f0bf78e1640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f0bf78e1640 10 monclient(hunting): authenticate will time out at 1257.260065s 2024-08-09T14:50:50.248 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7fac61bb3640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2024-08-09T14:50:50.248 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7fac61bb3640 1 -- 172.21.15.155:0/2895706432 >> v2:172.21.15.194:3300/0 conn(0x7fac5c0b26e0 msgr2=0x7fac5c0b3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.248 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7fac61bb3640 1 --2- 172.21.15.155:0/2895706432 >> v2:172.21.15.194:3300/0 conn(0x7fac5c0b26e0 0x7fac5c0b3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.248 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7fac61bb3640 1 -- 172.21.15.155:0/2895706432 >> v1:172.21.15.194:6789/0 conn(0x7fac5c0b22e0 legacy=0x7fac5c1469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.248 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7fac61bb3640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.248 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7fac61bb3640 1 monclient: found mon.noname-b 2024-08-09T14:50:50.248 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7fac61bb3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.248 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7fac61bb3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.248 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7fac61bb3640 1 -- 172.21.15.155:0/2895706432 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5c0b3f30 con 0x7fac5c0b0160 2024-08-09T14:50:50.249 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f94bca23640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.249 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f94bca23640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-08-09T14:50:50.249 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f94bca23640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94b806b910 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:50.250 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f94bca23640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f94b80affd0 0x7f94b8072be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.250 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f94bca23640 10 monclient(hunting): picked mon.noname-c con 0x7f94b80affd0 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:50.250 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f94bca23640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94b806bcf0 0x7f94b814d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.250 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f94bca23640 10 monclient(hunting): picked mon.noname-b con 0x7f94b806bcf0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.250 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f94bca23640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f94b8072710 con 0x7f94b806b910 2024-08-09T14:50:50.250 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f94bca23640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.250 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f94bca23640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f94b80b0480 con 0x7f94b80affd0 2024-08-09T14:50:50.250 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f94bca23640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.250 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f94bca23640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f94b80b1dd0 con 0x7f94b806bcf0 2024-08-09T14:50:50.250 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f94bca23640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.250 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f94bca23640 10 monclient(hunting): authenticate will time out at 1257.260540s 2024-08-09T14:50:50.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.223+0000 7f2481ccc640 1 --2- 172.21.15.155:0/3320873738 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f247c0aefc0 0x7f247c1534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.224+0000 7f2481ccc640 10 monclient(hunting): get_auth_request con 0x7f247c0aefc0 auth_method 0 2024-08-09T14:50:50.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.224+0000 7f2481ccc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.224+0000 7f2481ccc640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.224+0000 7f2481ccc640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.224+0000 7f3d2141f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.224+0000 7f3d2141f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.224+0000 7f3d2141f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.225+0000 7f2481ccc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.225+0000 7f2481ccc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.225+0000 7f2481ccc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.225+0000 7f94b6d9d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94b806bcf0 0x7f94b814d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.225+0000 7f0bf4e7c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bf00afb50 0x7f0bf013d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.226+0000 7fa696d9d640 1 --2- 172.21.15.155:0/3741380805 >> v2:172.21.15.194:3300/0 conn(0x7fa6981449a0 0x7fa698147570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.227+0000 7f2481ccc640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2024-08-09T14:50:50.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.227+0000 7f2481ccc640 1 -- 172.21.15.155:0/3320873738 >> v2:172.21.15.194:3300/0 conn(0x7f247c0ab520 msgr2=0x7f247c0aa5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.227+0000 7f2481ccc640 1 --2- 172.21.15.155:0/3320873738 >> v2:172.21.15.194:3300/0 conn(0x7f247c0ab520 0x7f247c0aa5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.227+0000 7f2481ccc640 1 -- 172.21.15.155:0/3320873738 >> v1:172.21.15.155:6789/0 conn(0x7f247c0af400 legacy=0x7f247c153a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.227+0000 7f2481ccc640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.227+0000 7f2481ccc640 1 monclient: found mon.noname-b 2024-08-09T14:50:50.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.227+0000 7f2481ccc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.227+0000 7f2481ccc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.227+0000 7f2481ccc640 1 -- 172.21.15.155:0/3320873738 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247c08cce0 con 0x7f247c0aefc0 2024-08-09T14:50:50.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.227+0000 7fa695d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.227+0000 7f6ca7fff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f6ca80aac00 0x7f6ca8151860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:34530/0 (socket says 172.21.15.155:34530) 2024-08-09T14:50:50.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.227+0000 7f6ca7fff640 1 -- 172.21.15.155:0/3003073810 learned_addr learned my addr 172.21.15.155:0/3003073810 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:50.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.227+0000 7fa695d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.227+0000 7fa695d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.228+0000 7f6ca77fe640 1 --2- 172.21.15.155:0/3003073810 >> v2:172.21.15.194:3300/0 conn(0x7f6ca80aa820 0x7f6ca8144290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.228+0000 7f94b6d9d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94b806bcf0 0x7f94b814d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54634/0 (socket says 172.21.15.155:54634) 2024-08-09T14:50:50.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.228+0000 7f94b6d9d640 1 -- 172.21.15.155:0/1175701238 learned_addr learned my addr 172.21.15.155:0/1175701238 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.228+0000 7f94b6d9d640 10 monclient(hunting): get_auth_request con 0x7f94b806bcf0 auth_method 0 2024-08-09T14:50:50.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.228+0000 7f94b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.228+0000 7f94b6d9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.228+0000 7f94b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.229+0000 7f6ca77fe640 10 monclient(hunting): get_auth_request con 0x7f6ca80aa820 auth_method 0 2024-08-09T14:50:50.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.229+0000 7f6ca77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.229+0000 7f6ca77fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.229+0000 7f6ca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.229+0000 7f94b6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.229+0000 7f94b6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.229+0000 7f94b6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.229+0000 7f0bf4e7c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bf00afb50 0x7f0bf013d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54622/0 (socket says 172.21.15.155:54622) 2024-08-09T14:50:50.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.229+0000 7f0bf4e7c640 1 -- 172.21.15.155:0/2409911611 learned_addr learned my addr 172.21.15.155:0/2409911611 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.229+0000 7f0bf4e7c640 10 monclient(hunting): get_auth_request con 0x7f0bf00afb50 auth_method 0 2024-08-09T14:50:50.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.229+0000 7f0bf4e7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.229+0000 7f0bf4e7c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.229+0000 7f0bf4e7c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.258 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.230+0000 7fa696d9d640 10 monclient(hunting): get_auth_request con 0x7fa6981449a0 auth_method 0 2024-08-09T14:50:50.258 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.230+0000 7fa696d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.258 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.230+0000 7fa696d9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.258 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.230+0000 7fa696d9d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.258 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.230+0000 7fa696d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.258 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.230+0000 7fa696d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.258 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.230+0000 7fa696d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.231+0000 7f6ca77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.231+0000 7f6ca77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.231+0000 7f6ca77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.232+0000 7f94b6d9d640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-08-09T14:50:50.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.232+0000 7f94b6d9d640 1 -- 172.21.15.155:0/1175701238 >> v2:172.21.15.194:3300/0 conn(0x7f94b80affd0 msgr2=0x7f94b8072be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.232+0000 7f94b6d9d640 1 --2- 172.21.15.155:0/1175701238 >> v2:172.21.15.194:3300/0 conn(0x7f94b80affd0 0x7f94b8072be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.232+0000 7f94b6d9d640 1 -- 172.21.15.155:0/1175701238 >> v1:172.21.15.155:6789/0 conn(0x7f94b806b910 legacy=0x7f94b8072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.232+0000 7f94b6d9d640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.232+0000 7f94b6d9d640 1 monclient: found mon.noname-b 2024-08-09T14:50:50.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.232+0000 7f94b6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.232+0000 7f94b6d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.232+0000 7f94b6d9d640 1 -- 172.21.15.155:0/1175701238 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b806c690 con 0x7f94b806bcf0 2024-08-09T14:50:50.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.232+0000 7f201edef640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.232+0000 7f201edef640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7f0bf4e7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.261 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7f0bf4e7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.261 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7f0bf4e7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.261 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.232+0000 7f201edef640 10 monclient: monmap: 2024-08-09T14:50:50.261 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.261 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.261 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.233981+0000 2024-08-09T14:50:50.261 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.233981+0000 2024-08-09T14:50:50.261 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.261 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.262 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.262 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.262 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.262 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.262 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.262 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7fa696d9d640 10 monclient(hunting): handle_auth_done global_id 4273 payload 274 2024-08-09T14:50:50.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7fa696d9d640 1 -- 172.21.15.155:0/3741380805 >> v2:172.21.15.155:3300/0 conn(0x7fa6980ae9c0 msgr2=0x7fa698147030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7fa696d9d640 1 --2- 172.21.15.155:0/3741380805 >> v2:172.21.15.155:3300/0 conn(0x7fa6980ae9c0 0x7fa698147030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7fa696d9d640 1 -- 172.21.15.155:0/3741380805 >> v1:172.21.15.155:6789/0 conn(0x7fa6980aa820 legacy=0x7fa698144290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7fa696d9d640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7fa696d9d640 1 monclient: found mon.noname-c 2024-08-09T14:50:50.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7fa696d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7fa696d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7fa696d9d640 1 -- 172.21.15.155:0/3741380805 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa698147ab0 con 0x7fa6981449a0 2024-08-09T14:50:50.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7f201edef640 1 Processor -- start 2024-08-09T14:50:50.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7f201edef640 1 -- start start 2024-08-09T14:50:50.264 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.233+0000 7f201edef640 10 monclient: init 2024-08-09T14:50:50.265 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.265 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-08-09T14:50:50.265 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20180b0160 0x7f20180b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.265 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 10 monclient(hunting): picked mon.noname-b con 0x7f20180b0160 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.265 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f20180b22e0 0x7f20181469d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.265 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 10 monclient(hunting): picked mon.noname-a con 0x7f20180b22e0 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:50.265 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f2018146f10 0x7f20180b3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.265 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 10 monclient(hunting): picked mon.noname-c con 0x7f2018146f10 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:50.265 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f2481ccc640 1 --2- 172.21.15.155:0/3320873738 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f247c0aefc0 0x7f247c1534e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f246c0097a0 tx=0x7f246c02dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62f3b84851c1ca8e server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f20181461c0 con 0x7f20180b22e0 2024-08-09T14:50:50.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f20180b26e0 con 0x7f2018146f10 2024-08-09T14:50:50.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f20180b2890 con 0x7f20180b0160 2024-08-09T14:50:50.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.266 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.234+0000 7f201edef640 10 monclient(hunting): authenticate will time out at 1257.271472s 2024-08-09T14:50:50.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.236+0000 7f0bf4e7c640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-08-09T14:50:50.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.236+0000 7f0bf4e7c640 1 -- 172.21.15.155:0/2409911611 >> v2:172.21.15.155:3300/0 conn(0x7f0bf0071010 msgr2=0x7f0bf013cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.236+0000 7f0bf4e7c640 1 --2- 172.21.15.155:0/2409911611 >> v2:172.21.15.155:3300/0 conn(0x7f0bf0071010 0x7f0bf013cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.236+0000 7f0bf4e7c640 1 -- 172.21.15.155:0/2409911611 >> v1:172.21.15.155:6789/0 conn(0x7f0bf00713f0 legacy=0x7f0bf0072e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.236+0000 7f0bf4e7c640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.236+0000 7f0bf4e7c640 1 monclient: found mon.noname-b 2024-08-09T14:50:50.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.236+0000 7f0bf4e7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.236+0000 7f0bf4e7c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.267 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.236+0000 7f0bf4e7c640 1 -- 172.21.15.155:0/2409911611 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf0073540 con 0x7f0bf00afb50 2024-08-09T14:50:50.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.238+0000 7f3d2141f640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-08-09T14:50:50.268 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.238+0000 7f3d2141f640 1 -- 172.21.15.155:0/3860018960 >> v1:172.21.15.194:6789/0 conn(0x7f3d14006510 legacy=0x7f3d14034480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.238+0000 7f3d2141f640 1 -- 172.21.15.155:0/3860018960 >> v1:172.21.15.155:6789/0 conn(0x7f3d140c7640 legacy=0x7f3d14031e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.238+0000 7f3d2141f640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.238+0000 7f3d2141f640 1 monclient: found mon.noname-a 2024-08-09T14:50:50.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.238+0000 7f3d2141f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.238+0000 7f3d2141f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.238+0000 7f3d2141f640 1 -- 172.21.15.155:0/3860018960 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d140cadf0 con 0x7f3d140c34e0 2024-08-09T14:50:50.269 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.239+0000 7fa696d9d640 1 --2- 172.21.15.155:0/3741380805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6981449a0 0x7fa698147570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa6980b0b20 tx=0x7fa68c02b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27566b5717e937cc server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.270 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.240+0000 7f24814cb640 1 -- 172.21.15.155:0/3320873738 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c02f8a0 con 0x7f247c0aefc0 2024-08-09T14:50:50.270 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.240+0000 7f24814cb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.270 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.240+0000 7f24814cb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:50.270 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.240+0000 7f24814cb640 10 monclient: dump: 2024-08-09T14:50:50.270 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.270 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.270 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.270 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.270 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.270 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.270 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.271 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.271 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.271 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.271 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.240+0000 7f24814cb640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.271 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.240+0000 7f24814cb640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.271 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.240+0000 7f24814cb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.271 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.240+0000 7f2484731640 5 monclient: authenticate success, global_id 4265 2024-08-09T14:50:50.271 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.240+0000 7f2484731640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:50.273 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.240+0000 7f201d38c640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f2018146f10 0x7f20180b3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.274 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.241+0000 7f0bf4e7c640 1 --2- 172.21.15.155:0/2409911611 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bf00afb50 0x7f0bf013d060 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0be0009870 tx=0x7f0be002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42d48ee0168f37e6 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.274 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.241+0000 7f94b6d9d640 1 --2- 172.21.15.155:0/1175701238 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94b806bcf0 0x7f94b814d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f94b0009e50 tx=0x7f94b002bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81104c4a25c23b52 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.274 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.244+0000 7f3d2141f640 1 --2- 172.21.15.155:0/3860018960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d140c34e0 0x7f3d140c7100 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009870 tx=0x7f3d1802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0116691acc1c8 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.275 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.245+0000 7f6ca77fe640 10 monclient(hunting): handle_auth_done global_id 4276 payload 274 2024-08-09T14:50:50.275 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.245+0000 7f6ca77fe640 1 -- 172.21.15.155:0/3003073810 >> v1:172.21.15.194:6789/0 conn(0x7f6ca80ae9c0 legacy=0x7f6ca8147030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.275 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.245+0000 7f6ca77fe640 1 -- 172.21.15.155:0/3003073810 >> v1:172.21.15.155:6789/0 conn(0x7f6ca80aac00 legacy=0x7f6ca8151860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.275 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.245+0000 7f6ca77fe640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.275 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.245+0000 7f6ca77fe640 1 monclient: found mon.noname-c 2024-08-09T14:50:50.275 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.245+0000 7f6ca77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.275 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.245+0000 7f6ca77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.245+0000 7f6ca77fe640 1 -- 172.21.15.155:0/3003073810 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca81447d0 con 0x7f6ca80aa820 2024-08-09T14:50:50.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.246+0000 7f201d38c640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f2018146f10 0x7f20180b3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:47792/0 (socket says 172.21.15.155:47792) 2024-08-09T14:50:50.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.246+0000 7f201d38c640 1 -- 172.21.15.155:0/2206401516 learned_addr learned my addr 172.21.15.155:0/2206401516 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.246+0000 7f201d38c640 10 monclient(hunting): get_auth_request con 0x7f2018146f10 auth_method 0 2024-08-09T14:50:50.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.246+0000 7f201d38c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.246+0000 7f201d38c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.276 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.246+0000 7f201d38c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.246+0000 7f201d38c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.246+0000 7f201d38c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.246+0000 7f201d38c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.248+0000 7f24814cb640 1 -- 172.21.15.155:0/3320873738 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f246c02fa40 con 0x7f247c0aefc0 2024-08-09T14:50:50.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.248+0000 7f24814cb640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.248+0000 7f24814cb640 1 -- 172.21.15.155:0/3320873738 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f246c02fe10 con 0x7f247c0aefc0 2024-08-09T14:50:50.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.248+0000 7f2484731640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:50.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.248+0000 7f2484731640 4 set_mon_vals no callback set 2024-08-09T14:50:50.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.248+0000 7f24814cb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.248+0000 7f24814cb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:50.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.248+0000 7f24814cb640 10 monclient: dump: 2024-08-09T14:50:50.278 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.279 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.279 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.279 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.279 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.279 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.279 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.279 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.279 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.279 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.249+0000 7f2484731640 10 monclient: shutdown 2024-08-09T14:50:50.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.249+0000 7f2484731640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.249+0000 7f2484731640 1 -- 172.21.15.155:0/3320873738 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f247c0aefc0 msgr2=0x7f247c1534e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.249+0000 7f2484731640 1 --2- 172.21.15.155:0/3320873738 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f247c0aefc0 0x7f247c1534e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f246c0097a0 tx=0x7f246c02dae0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.249+0000 7f2484731640 1 -- 172.21.15.155:0/3320873738 shutdown_connections 2024-08-09T14:50:50.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.249+0000 7f2484731640 1 --2- 172.21.15.155:0/3320873738 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f247c0aefc0 0x7f247c1534e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.249+0000 7f2484731640 1 --2- 172.21.15.155:0/3320873738 >> v2:172.21.15.194:3300/0 conn(0x7f247c0ab520 0x7f247c0aa5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.249+0000 7f2484731640 1 -- 172.21.15.155:0/3320873738 >> 172.21.15.155:0/3320873738 conn(0x7f247c06c200 msgr2=0x7f247c0b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.249+0000 7f2484731640 1 -- 172.21.15.155:0/3320873738 shutdown_connections 2024-08-09T14:50:50.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.249+0000 7f2484731640 1 -- 172.21.15.155:0/3320873738 wait complete. 2024-08-09T14:50:50.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f201d38c640 10 monclient(hunting): handle_auth_done global_id 4279 payload 274 2024-08-09T14:50:50.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f201d38c640 1 -- 172.21.15.155:0/2206401516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20180b0160 msgr2=0x7f20180b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f201d38c640 1 --2- 172.21.15.155:0/2206401516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20180b0160 0x7f20180b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f201d38c640 1 -- 172.21.15.155:0/2206401516 >> v2:172.21.15.155:3300/0 conn(0x7f20180b22e0 msgr2=0x7f20181469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f201d38c640 1 --2- 172.21.15.155:0/2206401516 >> v2:172.21.15.155:3300/0 conn(0x7f20180b22e0 0x7f20181469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f201d38c640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f201d38c640 1 monclient: found mon.noname-c 2024-08-09T14:50:50.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f201d38c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f201d38c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.281 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f201d38c640 1 -- 172.21.15.155:0/2206401516 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20180b3d80 con 0x7f2018146f10 2024-08-09T14:50:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f2484731640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f2484731640 10 monclient: monmap: 2024-08-09T14:50:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.282 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.283 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:50.283 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:50.283 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:50.283 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.283 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f2484731640 1 librados: starting msgr at 2024-08-09T14:50:50.283 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f2484731640 1 librados: starting objecter 2024-08-09T14:50:50.283 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0be7fff640 1 -- 172.21.15.155:0/2409911611 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be00317c0 con 0x7f0bf00afb50 2024-08-09T14:50:50.283 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0be7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0be7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:50.284 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0be7fff640 10 monclient: dump: 2024-08-09T14:50:50.284 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.284 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.284 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.284 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.284 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.284 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.284 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.284 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.284 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.284 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0be7fff640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0be7fff640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0be7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0be7fff640 1 -- 172.21.15.155:0/2409911611 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0be0037070 con 0x7f0bf00afb50 2024-08-09T14:50:50.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0be7fff640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0be7fff640 1 -- 172.21.15.155:0/2409911611 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be0031b80 con 0x7f0bf00afb50 2024-08-09T14:50:50.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0bf78e1640 5 monclient: authenticate success, global_id 4271 2024-08-09T14:50:50.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0bf78e1640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:50.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0bf78e1640 4 set_mon_vals no callback set 2024-08-09T14:50:50.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0be7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0be7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:50.285 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f0be7fff640 10 monclient: dump: 2024-08-09T14:50:50.285 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.286 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.286 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.286 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.286 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.286 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.286 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.286 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.286 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.286 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.250+0000 7f2484731640 1 Processor -- start 2024-08-09T14:50:50.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7f2484731640 1 -- start start 2024-08-09T14:50:50.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7f2484731640 1 librados: setting wanted keys 2024-08-09T14:50:50.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7f2484731640 1 librados: calling monclient init 2024-08-09T14:50:50.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7f2484731640 10 monclient: init 2024-08-09T14:50:50.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69559a640 1 -- 172.21.15.155:0/3741380805 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c038070 con 0x7fa6981449a0 2024-08-09T14:50:50.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:50.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69559a640 10 monclient: dump: 2024-08-09T14:50:50.287 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.288 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.288 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.288 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.288 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.288 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.288 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.288 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.288 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.288 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.288 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69559a640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.288 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69559a640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.288 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.288 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69559a640 1 -- 172.21.15.155:0/3741380805 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa68c031710 con 0x7fa6981449a0 2024-08-09T14:50:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69559a640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69559a640 1 -- 172.21.15.155:0/3741380805 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa68c031ae0 con 0x7fa6981449a0 2024-08-09T14:50:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69559a640 10 monclient: dump: 2024-08-09T14:50:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.289 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.290 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69cbed640 5 monclient: authenticate success, global_id 4273 2024-08-09T14:50:50.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69cbed640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:50.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7fa69cbed640 4 set_mon_vals no callback set 2024-08-09T14:50:50.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7f0bf78e1640 10 monclient: shutdown 2024-08-09T14:50:50.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7f0bf78e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2409911611 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bf00afb50 msgr2=0x7f0bf013d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.290 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7f0bf78e1640 1 --2- 172.21.15.155:0/2409911611 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bf00afb50 0x7f0bf013d060 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0be0009870 tx=0x7f0be002b860 comp rx=0 tx=0).stop 2024-08-09T14:50:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7f2484731640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7f2484731640 10 monclient: _add_conns ranks=[2,0,1] 2024-08-09T14:50:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.251+0000 7f2484731640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f247c0aefc0 0x7f247c072f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2409911611 shutdown_connections 2024-08-09T14:50:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f0bf78e1640 1 --2- 172.21.15.155:0/2409911611 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bf00afb50 0x7f0bf013d060 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f0bf78e1640 1 --2- 172.21.15.155:0/2409911611 >> v2:172.21.15.155:3300/0 conn(0x7f0bf0071010 0x7f0bf013cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2409911611 >> 172.21.15.155:0/2409911611 conn(0x7f0bf00b5af0 msgr2=0x7f0bf00b5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f2484731640 10 monclient(hunting): picked mon.noname-c con 0x7f247c0aefc0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f2484731640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f247c0af400 0x7f247c070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7fa69cbed640 10 monclient: shutdown 2024-08-09T14:50:50.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7fa69cbed640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7fa69cbed640 1 -- 172.21.15.155:0/3741380805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6981449a0 msgr2=0x7fa698147570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7fa69cbed640 1 --2- 172.21.15.155:0/3741380805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6981449a0 0x7fa698147570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa6980b0b20 tx=0x7fa68c02b7b0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f2484731640 10 monclient(hunting): picked mon.noname-a con 0x7f247c0af400 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:50.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f2484731640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f247c1f1200 0x7f247c1f1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7fa69cbed640 1 -- 172.21.15.155:0/3741380805 shutdown_connections 2024-08-09T14:50:50.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7fa69cbed640 1 --2- 172.21.15.155:0/3741380805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6981449a0 0x7fa698147570 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7fa69cbed640 1 --2- 172.21.15.155:0/3741380805 >> v2:172.21.15.155:3300/0 conn(0x7fa6980ae9c0 0x7fa698147030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7fa69cbed640 1 -- 172.21.15.155:0/3741380805 >> 172.21.15.155:0/3741380805 conn(0x7fa6980b45c0 msgr2=0x7fa6980b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.294 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f2484731640 10 monclient(hunting): picked mon.noname-b con 0x7f247c1f1200 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:50.294 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f2484731640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.294 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2409911611 shutdown_connections 2024-08-09T14:50:50.294 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2409911611 wait complete. 2024-08-09T14:50:50.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f2484731640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f247c0aadf0 con 0x7f247c0af400 2024-08-09T14:50:50.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f2484731640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f2484731640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f247c0aac70 con 0x7f247c1f1200 2024-08-09T14:50:50.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f2484731640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f2484731640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f247c0aaf70 con 0x7f247c0aefc0 2024-08-09T14:50:50.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f2484731640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.295 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7f2484731640 10 monclient(hunting): authenticate will time out at 1257.289384s 2024-08-09T14:50:50.296 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7fa69cbed640 1 -- 172.21.15.155:0/3741380805 shutdown_connections 2024-08-09T14:50:50.296 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7fa69cbed640 1 -- 172.21.15.155:0/3741380805 wait complete. 2024-08-09T14:50:50.296 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.252+0000 7fac61bb3640 1 --2- 172.21.15.155:0/2895706432 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac5c0b0160 0x7fac5c0b0540 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fac58009870 tx=0x7fac5802b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be869b72f47798dd server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.296 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7f0bf78e1640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.296 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7f0bf78e1640 10 monclient: monmap: 2024-08-09T14:50:50.296 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.296 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:50.296 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:50.297 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:50.297 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.297 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.297 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:50.297 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:50.297 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:50.297 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7fa69cbed640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.297 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7fa69cbed640 10 monclient: monmap: 2024-08-09T14:50:50.298 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.298 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:50.298 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:50.298 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:50.298 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.298 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.298 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:50.298 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:50.298 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:50.298 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7f0bf78e1640 1 librados: starting msgr at 2024-08-09T14:50:50.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7f0bf78e1640 1 librados: starting objecter 2024-08-09T14:50:50.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7f0bf78e1640 1 Processor -- start 2024-08-09T14:50:50.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7f0bf78e1640 1 -- start start 2024-08-09T14:50:50.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7f0bf78e1640 1 librados: setting wanted keys 2024-08-09T14:50:50.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7f0bf78e1640 1 librados: calling monclient init 2024-08-09T14:50:50.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7f0bf78e1640 10 monclient: init 2024-08-09T14:50:50.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7fa69cbed640 1 librados: starting msgr at 2024-08-09T14:50:50.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7fa69cbed640 1 librados: starting objecter 2024-08-09T14:50:50.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7f201d38c640 1 --2- 172.21.15.155:0/2206401516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2018146f10 0x7f20180b3840 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f20180b1db0 tx=0x7f201002bdb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3df4039dc6642be server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7fa69cbed640 1 Processor -- start 2024-08-09T14:50:50.300 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7fa69cbed640 1 -- start start 2024-08-09T14:50:50.301 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7fa69cbed640 1 librados: setting wanted keys 2024-08-09T14:50:50.301 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7fa69cbed640 1 librados: calling monclient init 2024-08-09T14:50:50.301 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.253+0000 7fa69cbed640 10 monclient: init 2024-08-09T14:50:50.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 10 monclient: _add_conns ranks=[2,0,1] 2024-08-09T14:50:50.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bf0071010 0x7f0bf0142c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 10 monclient(hunting): picked mon.noname-c con 0x7f0bf0071010 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bf00713f0 0x7f0bf01d7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 10 monclient(hunting): picked mon.noname-a con 0x7f0bf00713f0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:50.303 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0bf00afb50 0x7f0bf01d8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.303 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 10 monclient(hunting): picked mon.noname-b con 0x7f0bf00afb50 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:50.303 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.303 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf014a2e0 con 0x7f0bf00713f0 2024-08-09T14:50:50.303 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.303 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf014a160 con 0x7f0bf00afb50 2024-08-09T14:50:50.303 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.303 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf014a460 con 0x7f0bf0071010 2024-08-09T14:50:50.303 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.303 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7f0bf78e1640 10 monclient(hunting): authenticate will time out at 1257.291399s 2024-08-09T14:50:50.304 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.304 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 10 monclient: _add_conns ranks=[1,0,2] 2024-08-09T14:50:50.304 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6980aa820 0x7fa698134bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.304 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 10 monclient(hunting): picked mon.noname-b con 0x7fa6980aa820 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:50.304 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6980ae9c0 0x7fa698149950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.305 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 10 monclient(hunting): picked mon.noname-a con 0x7fa6980ae9c0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:50.305 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6981449a0 0x7fa698148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.305 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 10 monclient(hunting): picked mon.noname-c con 0x7fa6981449a0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.305 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.305 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa698151860 con 0x7fa6980ae9c0 2024-08-09T14:50:50.305 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.305 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa6980aae90 con 0x7fa6980aa820 2024-08-09T14:50:50.305 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.305 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fa6981519e0 con 0x7fa6981449a0 2024-08-09T14:50:50.305 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.305 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.254+0000 7fa69cbed640 10 monclient(hunting): authenticate will time out at 1257.291909s 2024-08-09T14:50:50.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.259+0000 7f6ca77fe640 1 --2- 172.21.15.155:0/3003073810 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ca80aa820 0x7f6ca8144290 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009d90 tx=0x7f6ca002bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70c91da1044daf79 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.259+0000 7f2482cce640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f247c1f1200 0x7f247c1f1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f24824cd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f247c0aefc0 0x7f247c072f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7fa69659c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6980aa820 0x7fa698134bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d20c1e640 1 -- 172.21.15.155:0/3860018960 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d18035020 con 0x7f3d140c34e0 2024-08-09T14:50:50.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d20c1e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d20c1e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:50.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d20c1e640 10 monclient: dump: 2024-08-09T14:50:50.307 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.307 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.308 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.308 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.308 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.308 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.308 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.308 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.308 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.308 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d20c1e640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d20c1e640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d20c1e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d20c1e640 1 -- 172.21.15.155:0/3860018960 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d18031870 con 0x7f3d140c34e0 2024-08-09T14:50:50.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d20c1e640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d20c1e640 1 -- 172.21.15.155:0/3860018960 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d18031c40 con 0x7f3d140c34e0 2024-08-09T14:50:50.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d22c22640 5 monclient: authenticate success, global_id 4293 2024-08-09T14:50:50.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d22c22640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:50.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d22c22640 4 set_mon_vals no callback set 2024-08-09T14:50:50.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d20c1e640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d20c1e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:50.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7f3d20c1e640 10 monclient: dump: 2024-08-09T14:50:50.309 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.309 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.309 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.309 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.309 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.309 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.310 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.310 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.310 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.310 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7fa69659c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6980aa820 0x7fa698134bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:47814/0 (socket says 172.21.15.155:47814) 2024-08-09T14:50:50.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7fa69659c640 1 -- 172.21.15.155:0/1179127270 learned_addr learned my addr 172.21.15.155:0/1179127270 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7fa69659c640 10 monclient(hunting): get_auth_request con 0x7fa6980aa820 auth_method 0 2024-08-09T14:50:50.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7fa69659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7fa69659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.260+0000 7fa69659c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f24824cd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f247c0aefc0 0x7f247c072f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54642/0 (socket says 172.21.15.155:54642) 2024-08-09T14:50:50.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f24824cd640 1 -- 172.21.15.155:0/1033534714 learned_addr learned my addr 172.21.15.155:0/1033534714 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f24824cd640 10 monclient(hunting): get_auth_request con 0x7f247c0aefc0 auth_method 0 2024-08-09T14:50:50.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f24824cd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f24824cd640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f24824cd640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f0bf567d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bf0071010 0x7f0bf0142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f2481ccc640 1 --2- 172.21.15.155:0/1033534714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f247c0af400 0x7f247c070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f3d22c22640 10 monclient: shutdown 2024-08-09T14:50:50.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f3d22c22640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f3d22c22640 1 -- 172.21.15.155:0/3860018960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d140c34e0 msgr2=0x7f3d140c7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f3d22c22640 1 --2- 172.21.15.155:0/3860018960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d140c34e0 0x7f3d140c7100 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009870 tx=0x7f3d1802b910 comp rx=0 tx=0).stop 2024-08-09T14:50:50.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f24824cd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f24824cd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f24824cd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f3d22c22640 1 -- 172.21.15.155:0/3860018960 shutdown_connections 2024-08-09T14:50:50.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f3d22c22640 1 --2- 172.21.15.155:0/3860018960 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d140c34e0 0x7f3d140c7100 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.261+0000 7f3d22c22640 1 -- 172.21.15.155:0/3860018960 >> 172.21.15.155:0/3860018960 conn(0x7f3d140059f0 msgr2=0x7f3d14005e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f3d22c22640 1 -- 172.21.15.155:0/3860018960 shutdown_connections 2024-08-09T14:50:50.313 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f3d22c22640 1 -- 172.21.15.155:0/3860018960 wait complete. 2024-08-09T14:50:50.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f2482cce640 10 monclient(hunting): get_auth_request con 0x7f247c1f1200 auth_method 0 2024-08-09T14:50:50.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f2482cce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f2482cce640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f2482cce640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7fa69659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.314 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7fa69659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7fa69659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f20177fe640 1 -- 172.21.15.155:0/2206401516 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201002c950 con 0x7f2018146f10 2024-08-09T14:50:50.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f20177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f20177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:50.315 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f20177fe640 10 monclient: dump: 2024-08-09T14:50:50.315 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.315 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.315 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.315 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.315 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.316 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.316 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.316 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.316 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.316 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f3d22c22640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.316 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f3d22c22640 10 monclient: monmap: 2024-08-09T14:50:50.316 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.316 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:50.316 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:50.316 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:50.316 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.317 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.317 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:50.317 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:50.317 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:50.317 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.317 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f20177fe640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.317 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f20177fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.317 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f20177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.317 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f20177fe640 1 -- 172.21.15.155:0/2206401516 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f201002caf0 con 0x7f2018146f10 2024-08-09T14:50:50.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f20177fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f20177fe640 1 -- 172.21.15.155:0/2206401516 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20100343f0 con 0x7f2018146f10 2024-08-09T14:50:50.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f20177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f20177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:50.318 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f20177fe640 10 monclient: dump: 2024-08-09T14:50:50.318 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.318 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.318 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.318 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.318 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.318 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.318 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.318 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.319 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.319 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.319 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f201edef640 5 monclient: authenticate success, global_id 4279 2024-08-09T14:50:50.319 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f201edef640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:50.319 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.262+0000 7f201edef640 4 set_mon_vals no callback set 2024-08-09T14:50:50.319 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f2482cce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.319 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f2482cce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.319 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f2482cce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 1 librados: starting msgr at 2024-08-09T14:50:50.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 1 librados: starting objecter 2024-08-09T14:50:50.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 1 Processor -- start 2024-08-09T14:50:50.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 1 -- start start 2024-08-09T14:50:50.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 1 librados: setting wanted keys 2024-08-09T14:50:50.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 1 librados: calling monclient init 2024-08-09T14:50:50.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 10 monclient: init 2024-08-09T14:50:50.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f2481ccc640 10 monclient(hunting): get_auth_request con 0x7f247c0af400 auth_method 0 2024-08-09T14:50:50.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f2481ccc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f2481ccc640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f2481ccc640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f201edef640 10 monclient: shutdown 2024-08-09T14:50:50.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f201edef640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f201edef640 1 -- 172.21.15.155:0/2206401516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2018146f10 msgr2=0x7f20180b3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f201edef640 1 --2- 172.21.15.155:0/2206401516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2018146f10 0x7f20180b3840 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f20180b1db0 tx=0x7f201002bdb0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.321 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.322 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 10 monclient: _add_conns ranks=[2,1,0] 2024-08-09T14:50:50.322 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f201edef640 1 -- 172.21.15.155:0/2206401516 shutdown_connections 2024-08-09T14:50:50.322 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f201edef640 1 --2- 172.21.15.155:0/2206401516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2018146f10 0x7f20180b3840 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.322 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f201edef640 1 --2- 172.21.15.155:0/2206401516 >> v2:172.21.15.155:3300/0 conn(0x7f20180b22e0 0x7f20181469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.322 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d14006510 0x7f3d140b65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.322 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 10 monclient(hunting): picked mon.noname-c con 0x7f3d14006510 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f201edef640 1 --2- 172.21.15.155:0/2206401516 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20180b0160 0x7f20180b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f201edef640 1 -- 172.21.15.155:0/2206401516 >> 172.21.15.155:0/2206401516 conn(0x7f201806dbd0 msgr2=0x7f20180676f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d140c34e0 0x7f3d140c4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 10 monclient(hunting): picked mon.noname-b con 0x7f3d140c34e0 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:50.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d140c7640 0x7f3d140c1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 10 monclient(hunting): picked mon.noname-a con 0x7f3d140c7640 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:50.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1402a3b0 con 0x7f3d140c7640 2024-08-09T14:50:50.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1402a230 con 0x7f3d140c34e0 2024-08-09T14:50:50.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f3d14032560 con 0x7f3d14006510 2024-08-09T14:50:50.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f3d22c22640 10 monclient(hunting): authenticate will time out at 1257.300935s 2024-08-09T14:50:50.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f201edef640 1 -- 172.21.15.155:0/2206401516 shutdown_connections 2024-08-09T14:50:50.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.263+0000 7f201edef640 1 -- 172.21.15.155:0/2206401516 wait complete. 2024-08-09T14:50:50.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.264+0000 7f2481ccc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.264+0000 7f2481ccc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.264+0000 7f2481ccc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.264+0000 7f0bf4e7c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bf00713f0 0x7f0bf01d7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.264+0000 7f201edef640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.325 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.264+0000 7f201edef640 10 monclient: monmap: 2024-08-09T14:50:50.325 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.326 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:50.326 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:50.326 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:50.326 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.326 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.326 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:50.326 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:50.326 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:50.326 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f201edef640 1 librados: starting msgr at 2024-08-09T14:50:50.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f201edef640 1 librados: starting objecter 2024-08-09T14:50:50.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f201edef640 1 Processor -- start 2024-08-09T14:50:50.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f201edef640 1 -- start start 2024-08-09T14:50:50.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f201edef640 1 librados: setting wanted keys 2024-08-09T14:50:50.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f201edef640 1 librados: calling monclient init 2024-08-09T14:50:50.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f201edef640 10 monclient: init 2024-08-09T14:50:50.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f24824cd640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2024-08-09T14:50:50.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f24824cd640 1 -- 172.21.15.155:0/1033534714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f247c1f1200 msgr2=0x7f247c1f1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f24824cd640 1 --2- 172.21.15.155:0/1033534714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f247c1f1200 0x7f247c1f1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f24824cd640 1 -- 172.21.15.155:0/1033534714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f247c0af400 msgr2=0x7f247c070bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f24824cd640 1 --2- 172.21.15.155:0/1033534714 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f247c0af400 0x7f247c070bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f24824cd640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f24824cd640 1 monclient: found mon.noname-c 2024-08-09T14:50:50.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f24824cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f24824cd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.265+0000 7f24824cd640 1 -- 172.21.15.155:0/1033534714 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f247c070950 con 0x7f247c0aefc0 2024-08-09T14:50:50.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 10 monclient: _add_conns ranks=[0,2,1] 2024-08-09T14:50:50.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20180b0160 0x7f20181ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 10 monclient(hunting): picked mon.noname-a con 0x7f20180b0160 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:50.329 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20180b22e0 0x7f20181d7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 10 monclient(hunting): picked mon.noname-c con 0x7f20180b22e0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2018146f10 0x7f20181d8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 10 monclient(hunting): picked mon.noname-b con 0x7f2018146f10 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:50.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f2018142cc0 con 0x7f20180b0160 2024-08-09T14:50:50.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f20180b4290 con 0x7f2018146f10 2024-08-09T14:50:50.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f2018142e40 con 0x7f20180b22e0 2024-08-09T14:50:50.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7f201edef640 10 monclient(hunting): authenticate will time out at 1257.303697s 2024-08-09T14:50:50.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7fa69659c640 10 monclient(hunting): handle_auth_done global_id 4282 payload 824 2024-08-09T14:50:50.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7fa69659c640 1 -- 172.21.15.155:0/1179127270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6981449a0 msgr2=0x7fa698148a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7fa69659c640 1 --2- 172.21.15.155:0/1179127270 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa6981449a0 0x7fa698148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7fa69659c640 1 -- 172.21.15.155:0/1179127270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6980ae9c0 msgr2=0x7fa698149950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7fa69659c640 1 --2- 172.21.15.155:0/1179127270 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa6980ae9c0 0x7fa698149950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7fa69659c640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7fa69659c640 1 monclient: found mon.noname-b 2024-08-09T14:50:50.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7fa69659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7fa69659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.266+0000 7fa69659c640 1 -- 172.21.15.155:0/1179127270 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa69814d9b0 con 0x7fa6980aa820 2024-08-09T14:50:50.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.268+0000 7f0bf567d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bf0071010 0x7f0bf0142c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54654/0 (socket says 172.21.15.155:54654) 2024-08-09T14:50:50.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.268+0000 7f0bf567d640 1 -- 172.21.15.155:0/2103563403 learned_addr learned my addr 172.21.15.155:0/2103563403 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.268+0000 7f0bf567d640 10 monclient(hunting): get_auth_request con 0x7f0bf0071010 auth_method 0 2024-08-09T14:50:50.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.268+0000 7f0bf567d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.268+0000 7f0bf567d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.268+0000 7f0bf567d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.269+0000 7fa69659c640 1 --2- 172.21.15.155:0/1179127270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6980aa820 0x7fa698134bd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa69000c870 tx=0x7fa69000cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.269+0000 7fa686ffd640 10 client.?.objecter ms_handle_connect 0x7fa6980aa820 2024-08-09T14:50:50.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.269+0000 7fa686ffd640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:50.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.269+0000 7f0bf567d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.269+0000 7f0bf567d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.269+0000 7f0bf567d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.334 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.269+0000 7f0bf4e7c640 10 monclient(hunting): get_auth_request con 0x7f0bf00713f0 auth_method 0 2024-08-09T14:50:50.334 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.269+0000 7f0bf4e7c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.334 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.269+0000 7f0bf4e7c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.334 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.269+0000 7f0bf4e7c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94b5d9b640 1 -- 172.21.15.155:0/1175701238 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b002c950 con 0x7f94b806bcf0 2024-08-09T14:50:50.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94b5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94b5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:50.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94b5d9b640 10 monclient: dump: 2024-08-09T14:50:50.335 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.335 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.335 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.335 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.335 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.335 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.336 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.336 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.336 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.336 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94b5d9b640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94b5d9b640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94b5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94b5d9b640 1 -- 172.21.15.155:0/1175701238 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94b002caf0 con 0x7f94b806bcf0 2024-08-09T14:50:50.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94bca23640 5 monclient: authenticate success, global_id 4268 2024-08-09T14:50:50.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94bca23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:50.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94b5d9b640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94b5d9b640 1 -- 172.21.15.155:0/1175701238 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94b00343f0 con 0x7f94b806bcf0 2024-08-09T14:50:50.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94bca23640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:50.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94bca23640 4 set_mon_vals no callback set 2024-08-09T14:50:50.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f24824cd640 1 --2- 172.21.15.155:0/1033534714 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f247c0aefc0 0x7f247c072f40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f247800e8a0 tx=0x7f247800ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f2472ffd640 10 client.?.objecter ms_handle_connect 0x7f247c0aefc0 2024-08-09T14:50:50.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f2472ffd640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:50.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94b5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94b5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:50.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.270+0000 7f94b5d9b640 10 monclient: dump: 2024-08-09T14:50:50.338 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.338 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.338 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.338 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.338 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.338 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.338 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.338 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.338 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.338 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.271+0000 7f94bca23640 10 monclient: shutdown 2024-08-09T14:50:50.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.271+0000 7f94bca23640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.271+0000 7f0bf4e7c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.271+0000 7f0bf4e7c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.339 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.271+0000 7f0bf4e7c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.271+0000 7f94bca23640 1 -- 172.21.15.155:0/1175701238 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94b806bcf0 msgr2=0x7f94b814d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.271+0000 7f94bca23640 1 --2- 172.21.15.155:0/1175701238 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94b806bcf0 0x7f94b814d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f94b0009e50 tx=0x7f94b002bdb0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.271+0000 7f94bca23640 1 -- 172.21.15.155:0/1175701238 shutdown_connections 2024-08-09T14:50:50.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.271+0000 7f94bca23640 1 --2- 172.21.15.155:0/1175701238 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94b806bcf0 0x7f94b814d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.271+0000 7f94bca23640 1 --2- 172.21.15.155:0/1175701238 >> v2:172.21.15.194:3300/0 conn(0x7f94b80affd0 0x7f94b8072be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.271+0000 7f94bca23640 1 -- 172.21.15.155:0/1175701238 >> 172.21.15.155:0/1175701238 conn(0x7f94b80b2700 msgr2=0x7f94b806ca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.271+0000 7f94bca23640 1 -- 172.21.15.155:0/1175701238 shutdown_connections 2024-08-09T14:50:50.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.271+0000 7f94bca23640 1 -- 172.21.15.155:0/1175701238 wait complete. 2024-08-09T14:50:50.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.272+0000 7f94bca23640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.272+0000 7f94bca23640 10 monclient: monmap: 2024-08-09T14:50:50.342 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.342 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:50.342 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:50.342 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:50.342 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.342 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.342 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:50.342 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:50.342 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:50.343 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.272+0000 7f94bca23640 1 librados: starting msgr at 2024-08-09T14:50:50.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.272+0000 7f94bca23640 1 librados: starting objecter 2024-08-09T14:50:50.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.272+0000 7f94bca23640 1 Processor -- start 2024-08-09T14:50:50.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.272+0000 7f94bca23640 1 -- start start 2024-08-09T14:50:50.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.272+0000 7f94bca23640 1 librados: setting wanted keys 2024-08-09T14:50:50.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.272+0000 7f94bca23640 1 librados: calling monclient init 2024-08-09T14:50:50.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.272+0000 7f94bca23640 10 monclient: init 2024-08-09T14:50:50.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 10 monclient: _add_conns ranks=[0,1,2] 2024-08-09T14:50:50.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94b806b910 0x7f94b81467f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 10 monclient(hunting): picked mon.noname-a con 0x7f94b806b910 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:50.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f94b806bcf0 0x7f94b81d87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 10 monclient(hunting): picked mon.noname-b con 0x7f94b806bcf0 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:50.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94b80affd0 0x7f94b81d8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 10 monclient(hunting): picked mon.noname-c con 0x7f94b80affd0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f94b8142be0 con 0x7f94b806b910 2024-08-09T14:50:50.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f94b8073370 con 0x7f94b806bcf0 2024-08-09T14:50:50.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f94b8142d60 con 0x7f94b80affd0 2024-08-09T14:50:50.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f94bca23640 10 monclient(hunting): authenticate will time out at 1257.310698s 2024-08-09T14:50:50.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f0bf567d640 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2024-08-09T14:50:50.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f0bf567d640 1 -- 172.21.15.155:0/2103563403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0bf00afb50 msgr2=0x7f0bf01d8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f0bf567d640 1 --2- 172.21.15.155:0/2103563403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0bf00afb50 0x7f0bf01d8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f0bf567d640 1 -- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bf00713f0 msgr2=0x7f0bf01d7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f0bf567d640 1 --2- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f0bf00713f0 0x7f0bf01d7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f0bf567d640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f0bf567d640 1 monclient: found mon.noname-c 2024-08-09T14:50:50.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f0bf567d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f0bf567d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.273+0000 7f0bf567d640 1 -- 172.21.15.155:0/2103563403 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf01464a0 con 0x7f0bf0071010 2024-08-09T14:50:50.346 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac60bb1640 1 -- 172.21.15.155:0/2895706432 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac580317c0 con 0x7fac5c0b0160 2024-08-09T14:50:50.346 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac60bb1640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.346 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac60bb1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:50.346 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac60bb1640 10 monclient: dump: 2024-08-09T14:50:50.346 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.347 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.347 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.347 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.347 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.347 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.347 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.347 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.347 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.347 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac60bb1640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac60bb1640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac60bb1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.347 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac60bb1640 1 -- 172.21.15.155:0/2895706432 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac58037070 con 0x7fac5c0b0160 2024-08-09T14:50:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac63e17640 5 monclient: authenticate success, global_id 4262 2024-08-09T14:50:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac63e17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac60bb1640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac60bb1640 1 -- 172.21.15.155:0/2895706432 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac58031b80 con 0x7fac5c0b0160 2024-08-09T14:50:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac63e17640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac63e17640 4 set_mon_vals no callback set 2024-08-09T14:50:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac60bb1640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac60bb1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7fac60bb1640 10 monclient: dump: 2024-08-09T14:50:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.348 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.349 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.349 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.349 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.349 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.349 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.349 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.276+0000 7f3d2141f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d140c34e0 0x7f3d140c4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.349 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7f3d2141f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d140c34e0 0x7f3d140c4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:47820/0 (socket says 172.21.15.155:47820) 2024-08-09T14:50:50.349 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7f3d2141f640 1 -- 172.21.15.155:0/4090729798 learned_addr learned my addr 172.21.15.155:0/4090729798 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.350 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7f3d2141f640 10 monclient(hunting): get_auth_request con 0x7f3d140c34e0 auth_method 0 2024-08-09T14:50:50.350 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7f3d2141f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.350 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7f3d2141f640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.350 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7f3d2141f640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.351 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7fac63e17640 10 monclient: shutdown 2024-08-09T14:50:50.351 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7fac63e17640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.351 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7fac63e17640 1 -- 172.21.15.155:0/2895706432 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac5c0b0160 msgr2=0x7fac5c0b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.351 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7fac63e17640 1 --2- 172.21.15.155:0/2895706432 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac5c0b0160 0x7fac5c0b0540 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fac58009870 tx=0x7fac5802b860 comp rx=0 tx=0).stop 2024-08-09T14:50:50.351 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7fac63e17640 1 -- 172.21.15.155:0/2895706432 shutdown_connections 2024-08-09T14:50:50.351 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7fac63e17640 1 --2- 172.21.15.155:0/2895706432 >> v2:172.21.15.194:3300/0 conn(0x7fac5c0b26e0 0x7fac5c0b3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.351 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7fac63e17640 1 --2- 172.21.15.155:0/2895706432 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac5c0b0160 0x7fac5c0b0540 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7fac63e17640 1 -- 172.21.15.155:0/2895706432 >> 172.21.15.155:0/2895706432 conn(0x7fac5c06dbd0 msgr2=0x7fac5c0676f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7fac63e17640 1 -- 172.21.15.155:0/2895706432 shutdown_connections 2024-08-09T14:50:50.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.277+0000 7fac63e17640 1 -- 172.21.15.155:0/2895706432 wait complete. 2024-08-09T14:50:50.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.278+0000 7f3d2141f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.278+0000 7f3d2141f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.278+0000 7f3d2141f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.353 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.278+0000 7fac63e17640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.353 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.278+0000 7fac63e17640 10 monclient: monmap: 2024-08-09T14:50:50.353 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.353 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:50.353 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:50.353 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:50.353 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.353 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.353 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:50.353 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:50.353 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:50.353 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.278+0000 7f3d22421640 1 --2- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d140c7640 0x7f3d140c1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.278+0000 7fac63e17640 1 librados: starting msgr at 2024-08-09T14:50:50.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.278+0000 7fac63e17640 1 librados: starting objecter 2024-08-09T14:50:50.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7fac63e17640 1 Processor -- start 2024-08-09T14:50:50.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7fac63e17640 1 -- start start 2024-08-09T14:50:50.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7fac63e17640 1 librados: setting wanted keys 2024-08-09T14:50:50.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7fac63e17640 1 librados: calling monclient init 2024-08-09T14:50:50.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7fac63e17640 10 monclient: init 2024-08-09T14:50:50.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7f3d22421640 10 monclient(hunting): get_auth_request con 0x7f3d140c7640 auth_method 0 2024-08-09T14:50:50.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7f3d22421640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7f3d22421640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7f3d22421640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7fac63e17640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7fac63e17640 10 monclient: _add_conns ranks=[2,1,0] 2024-08-09T14:50:50.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7fac63e17640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac5c0b0160 0x7fac5c14bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7fac63e17640 10 monclient(hunting): picked mon.noname-c con 0x7fac5c0b0160 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7fac63e17640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac5c0b22e0 0x7fac5c0aa920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.279+0000 7fac63e17640 10 monclient(hunting): picked mon.noname-b con 0x7fac5c0b22e0 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:50.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7fac63e17640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac5c0b26e0 0x7fac5c1495d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7fac63e17640 10 monclient(hunting): picked mon.noname-a con 0x7fac5c0b26e0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:50.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7fac63e17640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7fac63e17640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c14d6d0 con 0x7fac5c0b26e0 2024-08-09T14:50:50.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7fac63e17640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7fac63e17640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fac5c14d550 con 0x7fac5c0b22e0 2024-08-09T14:50:50.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7fac63e17640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7fac63e17640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fac5c14d850 con 0x7fac5c0b0160 2024-08-09T14:50:50.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7fac63e17640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7fac63e17640 10 monclient(hunting): authenticate will time out at 1257.317182s 2024-08-09T14:50:50.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7f94b659c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94b806b910 0x7f94b81467f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7f201cb8b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20180b0160 0x7f20181ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7f3d22421640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7f3d22421640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.280+0000 7f3d22421640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f201cb8b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20180b0160 0x7f20181ce4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43506/0 (socket says 172.21.15.155:43506) 2024-08-09T14:50:50.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f201cb8b640 1 -- 172.21.15.155:0/4032070454 learned_addr learned my addr 172.21.15.155:0/4032070454 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f201cb8b640 10 monclient(hunting): get_auth_request con 0x7f20180b0160 auth_method 0 2024-08-09T14:50:50.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f201cb8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f201cb8b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f201cb8b640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f3d2141f640 10 monclient(hunting): handle_auth_done global_id 4288 payload 824 2024-08-09T14:50:50.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f3d2141f640 1 -- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d14006510 msgr2=0x7f3d140b65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f3d2141f640 1 --2- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3d14006510 0x7f3d140b65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f3d2141f640 1 -- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d140c7640 msgr2=0x7f3d140c1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f3d2141f640 1 --2- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3d140c7640 0x7f3d140c1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f3d2141f640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f3d2141f640 1 monclient: found mon.noname-b 2024-08-09T14:50:50.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f3d2141f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f3d2141f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f3d2141f640 1 -- 172.21.15.155:0/4090729798 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d140c5680 con 0x7f3d140c34e0 2024-08-09T14:50:50.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2472ffd640 1 -- 172.21.15.155:0/1033534714 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2478017070 con 0x7f247c0aefc0 2024-08-09T14:50:50.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2472ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2472ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:50.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2472ffd640 10 monclient: dump: 2024-08-09T14:50:50.363 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.363 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.363 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.363 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.363 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.364 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.364 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.364 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.364 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.364 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2472ffd640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2472ffd640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2472ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2472ffd640 1 -- 172.21.15.155:0/1033534714 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f247800c940 con 0x7f247c0aefc0 2024-08-09T14:50:50.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2472ffd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2472ffd640 1 -- 172.21.15.155:0/1033534714 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f247800cd30 con 0x7f247c0aefc0 2024-08-09T14:50:50.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2472ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2472ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:50.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2472ffd640 10 monclient: dump: 2024-08-09T14:50:50.365 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.365 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.365 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.365 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.365 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.365 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.365 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.365 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.365 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.365 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2484731640 5 monclient: authenticate success, global_id 4274 2024-08-09T14:50:50.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2484731640 10 monclient: _renew_subs 2024-08-09T14:50:50.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2484731640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2484731640 1 -- 172.21.15.155:0/1033534714 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f247c072560 con 0x7f247c0aefc0 2024-08-09T14:50:50.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2480cca640 4 set_mon_vals no callback set 2024-08-09T14:50:50.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2484731640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2484731640 10 monclient: _renew_subs 2024-08-09T14:50:50.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2484731640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2484731640 1 -- 172.21.15.155:0/1033534714 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f247c1f1bd0 con 0x7f247c0aefc0 2024-08-09T14:50:50.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.281+0000 7f2484731640 1 librados: init done 2024-08-09T14:50:50.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.282+0000 7f0bf567d640 1 --2- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bf0071010 0x7f0bf0142c80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00b4a0 tx=0x7f0bec00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.282+0000 7f0be5ffb640 10 client.?.objecter ms_handle_connect 0x7f0bf0071010 2024-08-09T14:50:50.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.282+0000 7f0be5ffb640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:50.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.282+0000 7f2484731640 10 client.4274.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.282+0000 7f2484731640 15 client.4274.objecter _session_command_op_assign -1 1 2024-08-09T14:50:50.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.282+0000 7f2484731640 20 client.4274.objecter _get_session osd=-1 returning homeless 2024-08-09T14:50:50.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.282+0000 7f2484731640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.282+0000 7f2484731640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.282+0000 7f2484731640 1 -- 172.21.15.155:0/1033534714 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f247c0aa5b0 con 0x7f247c0aefc0 2024-08-09T14:50:50.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.284+0000 7f3d2141f640 1 --2- 172.21.15.155:0/4090729798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d140c34e0 0x7f3d140c4e40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009870 tx=0x7f3d180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.284+0000 7f3d127fc640 10 client.?.objecter ms_handle_connect 0x7f3d140c34e0 2024-08-09T14:50:50.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.284+0000 7f3d127fc640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:50.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.285+0000 7f201d38c640 1 --2- 172.21.15.155:0/4032070454 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2018146f10 0x7f20181d8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.285+0000 7f201d38c640 10 monclient(hunting): get_auth_request con 0x7f2018146f10 auth_method 0 2024-08-09T14:50:50.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.285+0000 7f201d38c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.285+0000 7f201d38c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.285+0000 7f201d38c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.286+0000 7f201cb8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.286+0000 7f201cb8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.286+0000 7f201cb8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.287+0000 7f94b659c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94b806b910 0x7f94b81467f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43522/0 (socket says 172.21.15.155:43522) 2024-08-09T14:50:50.369 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.287+0000 7f94b659c640 1 -- 172.21.15.155:0/976787639 learned_addr learned my addr 172.21.15.155:0/976787639 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.287+0000 7f94b659c640 10 monclient(hunting): get_auth_request con 0x7f94b806b910 auth_method 0 2024-08-09T14:50:50.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.287+0000 7f94b659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.287+0000 7f94b659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.287+0000 7f94b659c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.288+0000 7f2017fff640 1 --2- 172.21.15.155:0/4032070454 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20180b22e0 0x7f20181d7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.370 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f201d38c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f201d38c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f201d38c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f2017fff640 10 monclient(hunting): get_auth_request con 0x7f20180b22e0 auth_method 0 2024-08-09T14:50:50.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f2017fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f2017fff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f2017fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.371 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f94b6d9d640 1 --2- 172.21.15.155:0/976787639 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94b80affd0 0x7f94b81d8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6ca67fc640 1 -- 172.21.15.155:0/3003073810 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0039070 con 0x7f6ca80aa820 2024-08-09T14:50:50.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6ca67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6ca67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:50.372 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6ca67fc640 10 monclient: dump: 2024-08-09T14:50:50.372 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.372 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.372 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.372 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.372 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.372 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.372 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.372 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.373 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.373 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6ca67fc640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6ca67fc640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6ca67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6ca67fc640 1 -- 172.21.15.155:0/3003073810 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ca002cb40 con 0x7f6ca80aa820 2024-08-09T14:50:50.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6ca67fc640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6ca67fc640 1 -- 172.21.15.155:0/3003073810 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ca0034440 con 0x7f6ca80aa820 2024-08-09T14:50:50.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6cae0a0640 5 monclient: authenticate success, global_id 4276 2024-08-09T14:50:50.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6cae0a0640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:50.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6cae0a0640 4 set_mon_vals no callback set 2024-08-09T14:50:50.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6ca67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6ca67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:50.373 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.289+0000 7f6ca67fc640 10 monclient: dump: 2024-08-09T14:50:50.374 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.374 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.374 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.374 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.374 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.374 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.374 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.374 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.374 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.374 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.374 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f2017fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f2017fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f2017fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f6cae0a0640 10 monclient: shutdown 2024-08-09T14:50:50.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f6cae0a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f6cae0a0640 1 -- 172.21.15.155:0/3003073810 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ca80aa820 msgr2=0x7f6ca8144290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f6cae0a0640 1 --2- 172.21.15.155:0/3003073810 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ca80aa820 0x7f6ca8144290 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0009d90 tx=0x7f6ca002bd90 comp rx=0 tx=0).stop 2024-08-09T14:50:50.375 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f201cb8b640 10 monclient(hunting): handle_auth_done global_id 4317 payload 824 2024-08-09T14:50:50.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f6cae0a0640 1 -- 172.21.15.155:0/3003073810 shutdown_connections 2024-08-09T14:50:50.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f6cae0a0640 1 --2- 172.21.15.155:0/3003073810 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ca80aa820 0x7f6ca8144290 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f6cae0a0640 1 -- 172.21.15.155:0/3003073810 >> 172.21.15.155:0/3003073810 conn(0x7f6ca80b45c0 msgr2=0x7f6ca80b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f201cb8b640 1 -- 172.21.15.155:0/4032070454 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20180b22e0 msgr2=0x7f20181d7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f201cb8b640 1 --2- 172.21.15.155:0/4032070454 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f20180b22e0 0x7f20181d7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f201cb8b640 1 -- 172.21.15.155:0/4032070454 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2018146f10 msgr2=0x7f20181d8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.376 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f201cb8b640 1 --2- 172.21.15.155:0/4032070454 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2018146f10 0x7f20181d8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f201cb8b640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f201cb8b640 1 monclient: found mon.noname-a 2024-08-09T14:50:50.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f201cb8b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f201cb8b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f201cb8b640 1 -- 172.21.15.155:0/4032070454 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20180ab190 con 0x7f20180b0160 2024-08-09T14:50:50.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f6cae0a0640 1 -- 172.21.15.155:0/3003073810 shutdown_connections 2024-08-09T14:50:50.377 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f6cae0a0640 1 -- 172.21.15.155:0/3003073810 wait complete. 2024-08-09T14:50:50.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f94b6d9d640 10 monclient(hunting): get_auth_request con 0x7f94b80affd0 auth_method 0 2024-08-09T14:50:50.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f94b6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f94b6d9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.290+0000 7f94b6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.291+0000 7f94b659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.291+0000 7f94b659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.291+0000 7f94b659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.291+0000 7f6cae0a0640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.379 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.291+0000 7f6cae0a0640 10 monclient: monmap: 2024-08-09T14:50:50.379 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.380 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:50.380 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:50.380 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:50.380 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.380 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.380 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:50.380 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:50.380 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:50.380 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.380 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.291+0000 7f6cae0a0640 1 librados: starting msgr at 2024-08-09T14:50:50.380 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.291+0000 7f6cae0a0640 1 librados: starting objecter 2024-08-09T14:50:50.380 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.291+0000 7f6cae0a0640 1 Processor -- start 2024-08-09T14:50:50.380 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 1 -- start start 2024-08-09T14:50:50.381 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 1 librados: setting wanted keys 2024-08-09T14:50:50.381 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 1 librados: calling monclient init 2024-08-09T14:50:50.381 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 10 monclient: init 2024-08-09T14:50:50.381 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.381 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 10 monclient: _add_conns ranks=[0,2,1] 2024-08-09T14:50:50.381 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ca80aa820 0x7f6ca81d26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.381 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 10 monclient(hunting): picked mon.noname-a con 0x7f6ca80aa820 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:50.381 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ca80aac00 0x7f6ca8148330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.381 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 10 monclient(hunting): picked mon.noname-c con 0x7f6ca80aac00 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.381 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ca80ae9c0 0x7f6ca81dc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.381 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 10 monclient(hunting): picked mon.noname-b con 0x7f6ca80ae9c0 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:50.381 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.381 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca81478c0 con 0x7f6ca80aa820 2024-08-09T14:50:50.382 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.382 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca8147740 con 0x7f6ca80ae9c0 2024-08-09T14:50:50.382 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.382 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca8147a40 con 0x7f6ca80aac00 2024-08-09T14:50:50.382 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.382 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.292+0000 7f6cae0a0640 10 monclient(hunting): authenticate will time out at 1257.329663s 2024-08-09T14:50:50.383 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa686ffd640 1 -- 172.21.15.155:0/1179127270 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa690003b70 con 0x7fa6980aa820 2024-08-09T14:50:50.383 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa686ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.383 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa686ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:50.383 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa686ffd640 10 monclient: dump: 2024-08-09T14:50:50.383 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.383 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.383 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.383 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.384 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.384 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.384 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.384 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.384 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.384 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa686ffd640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa686ffd640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa686ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa686ffd640 1 -- 172.21.15.155:0/1179127270 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa690003d10 con 0x7fa6980aa820 2024-08-09T14:50:50.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa686ffd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa686ffd640 1 -- 172.21.15.155:0/1179127270 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa69000aa90 con 0x7fa6980aa820 2024-08-09T14:50:50.384 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa686ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa686ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:50.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa686ffd640 10 monclient: dump: 2024-08-09T14:50:50.385 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.385 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.385 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.385 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.385 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.385 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.385 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.385 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.385 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.385 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.385 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa69cbed640 5 monclient: authenticate success, global_id 4282 2024-08-09T14:50:50.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa69cbed640 10 monclient: _renew_subs 2024-08-09T14:50:50.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa69cbed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa69cbed640 1 -- 172.21.15.155:0/1179127270 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa69814dc90 con 0x7fa6980aa820 2024-08-09T14:50:50.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa694d99640 4 set_mon_vals no callback set 2024-08-09T14:50:50.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa69cbed640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa69cbed640 10 monclient: _renew_subs 2024-08-09T14:50:50.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa69cbed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa69cbed640 1 -- 172.21.15.155:0/1179127270 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa69814b740 con 0x7fa6980aa820 2024-08-09T14:50:50.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7fa69cbed640 1 librados: init done 2024-08-09T14:50:50.386 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.295+0000 7f94b659c640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-08-09T14:50:50.387 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7f94b659c640 1 -- 172.21.15.155:0/976787639 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94b80affd0 msgr2=0x7f94b81d8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.387 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7f94b659c640 1 --2- 172.21.15.155:0/976787639 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f94b80affd0 0x7f94b81d8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.387 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7f94b659c640 1 -- 172.21.15.155:0/976787639 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f94b806bcf0 msgr2=0x7f94b81d87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.387 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7f94b659c640 1 --2- 172.21.15.155:0/976787639 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f94b806bcf0 0x7f94b81d87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.387 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7f94b659c640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.387 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7f94b659c640 1 monclient: found mon.noname-a 2024-08-09T14:50:50.387 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7f94b659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.387 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7f94b659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.387 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7f94b659c640 1 -- 172.21.15.155:0/976787639 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94b8145830 con 0x7f94b806b910 2024-08-09T14:50:50.388 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7fa69cbed640 10 client.4282.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.388 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7fa69cbed640 15 client.4282.objecter _session_command_op_assign -1 1 2024-08-09T14:50:50.388 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7fa69cbed640 20 client.4282.objecter _get_session osd=-1 returning homeless 2024-08-09T14:50:50.388 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7fa69cbed640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.388 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7fa69cbed640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.388 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.296+0000 7fa69cbed640 1 -- 172.21.15.155:0/1179127270 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa6980ae4e0 con 0x7fa6980aa820 2024-08-09T14:50:50.388 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.298+0000 7f201cb8b640 1 --2- 172.21.15.155:0/4032070454 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20180b0160 0x7f20181ce4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f200c00b4a0 tx=0x7f200c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.388 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.298+0000 7f20157fa640 10 client.?.objecter ms_handle_connect 0x7f20180b0160 2024-08-09T14:50:50.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.298+0000 7f20157fa640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:50.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.299+0000 7fac61bb3640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac5c0b0160 0x7fac5c14bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.300+0000 7f94b659c640 1 --2- 172.21.15.155:0/976787639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94b806b910 0x7f94b81467f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f94a800b410 tx=0x7f94a800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.300+0000 7f94aeffd640 10 client.?.objecter ms_handle_connect 0x7f94b806b910 2024-08-09T14:50:50.389 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.300+0000 7f94aeffd640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:50.390 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0be5ffb640 1 -- 172.21.15.155:0/2103563403 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bec014070 con 0x7f0bf0071010 2024-08-09T14:50:50.390 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0be5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.390 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0be5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:50.390 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0be5ffb640 10 monclient: dump: 2024-08-09T14:50:50.390 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.391 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.391 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.391 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.391 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.391 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.391 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.391 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.391 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.391 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.391 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0be5ffb640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.391 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0be5ffb640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.391 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0be5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.391 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0be5ffb640 1 -- 172.21.15.155:0/2103563403 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bec010610 con 0x7f0bf0071010 2024-08-09T14:50:50.391 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0bf78e1640 5 monclient: authenticate success, global_id 4277 2024-08-09T14:50:50.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0bf78e1640 10 monclient: _renew_subs 2024-08-09T14:50:50.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0bf78e1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2103563403 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf0146150 con 0x7f0bf0071010 2024-08-09T14:50:50.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0be5ffb640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.392 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0be5ffb640 1 -- 172.21.15.155:0/2103563403 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bec010a00 con 0x7f0bf0071010 2024-08-09T14:50:50.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7fac61bb3640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac5c0b0160 0x7fac5c14bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54684/0 (socket says 172.21.15.155:54684) 2024-08-09T14:50:50.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7fac61bb3640 1 -- 172.21.15.155:0/2011517931 learned_addr learned my addr 172.21.15.155:0/2011517931 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7fac61bb3640 10 monclient(hunting): get_auth_request con 0x7fac5c0b0160 auth_method 0 2024-08-09T14:50:50.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7fac61bb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7fac61bb3640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7fac61bb3640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.393 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0be5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.394 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0be5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:50.394 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0be5ffb640 10 monclient: dump: 2024-08-09T14:50:50.394 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.394 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.394 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.394 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.394 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.394 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.394 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.394 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.394 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.394 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.395 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0bf78e1640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.395 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0bf78e1640 10 monclient: _renew_subs 2024-08-09T14:50:50.395 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0bf78e1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.395 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2103563403 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf01469c0 con 0x7f0bf0071010 2024-08-09T14:50:50.395 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0bf78e1640 1 librados: init done 2024-08-09T14:50:50.395 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f0be77fe640 4 set_mon_vals no callback set 2024-08-09T14:50:50.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f20157fa640 1 -- 172.21.15.155:0/4032070454 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200c015020 con 0x7f20180b0160 2024-08-09T14:50:50.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f20157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f20157fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:50.396 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f20157fa640 10 monclient: dump: 2024-08-09T14:50:50.397 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.397 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.397 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.397 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.397 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.397 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.397 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.397 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.397 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.397 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f20157fa640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f20157fa640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.397 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f20157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f20157fa640 1 -- 172.21.15.155:0/4032070454 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f200c011830 con 0x7f20180b0160 2024-08-09T14:50:50.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f20157fa640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f20157fa640 1 -- 172.21.15.155:0/4032070454 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f200c011c20 con 0x7f20180b0160 2024-08-09T14:50:50.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f20157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f20157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:50.398 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f20157fa640 10 monclient: dump: 2024-08-09T14:50:50.398 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.398 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.398 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.398 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.398 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.398 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.398 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.399 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.399 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.399 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.399 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f201edef640 5 monclient: authenticate success, global_id 4317 2024-08-09T14:50:50.399 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.304+0000 7f2016ffd640 4 set_mon_vals no callback set 2024-08-09T14:50:50.399 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f201edef640 10 monclient: _renew_subs 2024-08-09T14:50:50.399 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f201edef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.399 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f201edef640 1 -- 172.21.15.155:0/4032070454 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20180b9180 con 0x7f20180b0160 2024-08-09T14:50:50.399 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f201edef640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.399 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f201edef640 10 monclient: _renew_subs 2024-08-09T14:50:50.399 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f201edef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.399 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f201edef640 1 -- 172.21.15.155:0/4032070454 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20180b9470 con 0x7f20180b0160 2024-08-09T14:50:50.399 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f201edef640 1 librados: init done 2024-08-09T14:50:50.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f0bf78e1640 10 client.4277.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f0bf78e1640 15 client.4277.objecter _session_command_op_assign -1 1 2024-08-09T14:50:50.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f0bf78e1640 20 client.4277.objecter _get_session osd=-1 returning homeless 2024-08-09T14:50:50.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f0bf78e1640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f0bf78e1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.305+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2103563403 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0bf0146bf0 con 0x7f0bf0071010 2024-08-09T14:50:50.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.306+0000 7f201edef640 10 client.4317.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.306+0000 7f201edef640 15 client.4317.objecter _session_command_op_assign -1 1 2024-08-09T14:50:50.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.306+0000 7f201edef640 20 client.4317.objecter _get_session osd=-1 returning homeless 2024-08-09T14:50:50.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.306+0000 7f201edef640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.306+0000 7f201edef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.401 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.306+0000 7f201edef640 1 -- 172.21.15.155:0/4032070454 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f20181469d0 con 0x7f20180b0160 2024-08-09T14:50:50.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7fac623b4640 1 --2- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac5c0b26e0 0x7fac5c1495d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7fac61bb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7fac61bb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.402 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7fac61bb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d127fc640 1 -- 172.21.15.155:0/4090729798 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d18040070 con 0x7f3d140c34e0 2024-08-09T14:50:50.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d127fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:50.403 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d127fc640 10 monclient: dump: 2024-08-09T14:50:50.403 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.403 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.404 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.404 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.404 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.404 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.404 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.404 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.404 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.404 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.404 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d127fc640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.404 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d127fc640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.404 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.404 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d127fc640 1 -- 172.21.15.155:0/4090729798 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d18030e50 con 0x7f3d140c34e0 2024-08-09T14:50:50.404 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d127fc640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.405 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d127fc640 1 -- 172.21.15.155:0/4090729798 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d18035040 con 0x7f3d140c34e0 2024-08-09T14:50:50.405 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.405 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:50.405 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d127fc640 10 monclient: dump: 2024-08-09T14:50:50.405 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.405 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.405 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.405 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.405 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.405 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.405 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.405 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.405 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.405 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d13fff640 4 set_mon_vals no callback set 2024-08-09T14:50:50.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d22c22640 5 monclient: authenticate success, global_id 4288 2024-08-09T14:50:50.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d22c22640 10 monclient: _renew_subs 2024-08-09T14:50:50.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d22c22640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d22c22640 1 -- 172.21.15.155:0/4090729798 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d140c7bb0 con 0x7f3d140c34e0 2024-08-09T14:50:50.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d22c22640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d22c22640 10 monclient: _renew_subs 2024-08-09T14:50:50.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d22c22640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d22c22640 1 -- 172.21.15.155:0/4090729798 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d140c1bb0 con 0x7f3d140c34e0 2024-08-09T14:50:50.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f3d22c22640 1 librados: init done 2024-08-09T14:50:50.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.307+0000 7f6ca77fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ca80aa820 0x7f6ca81d26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.308+0000 7f6ca7fff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ca80ae9c0 0x7f6ca81dc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.308+0000 7f3d22c22640 10 client.4288.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.308+0000 7f3d22c22640 15 client.4288.objecter _session_command_op_assign -1 1 2024-08-09T14:50:50.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.308+0000 7f3d22c22640 20 client.4288.objecter _get_session osd=-1 returning homeless 2024-08-09T14:50:50.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.308+0000 7f3d22c22640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.308+0000 7f3d22c22640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.308+0000 7f3d22c22640 1 -- 172.21.15.155:0/4090729798 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3d140c7100 con 0x7f3d140c34e0 2024-08-09T14:50:50.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7fac613b2640 1 --2- 172.21.15.155:0/2011517931 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac5c0b22e0 0x7fac5c0aa920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7fac623b4640 10 monclient(hunting): get_auth_request con 0x7fac5c0b26e0 auth_method 0 2024-08-09T14:50:50.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7fac623b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.409 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7fac623b4640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.409 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7fac623b4640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.410 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94aeffd640 1 -- 172.21.15.155:0/976787639 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8015020 con 0x7f94b806b910 2024-08-09T14:50:50.410 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.410 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94aeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:50.410 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94aeffd640 10 monclient: dump: 2024-08-09T14:50:50.410 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.410 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.410 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.410 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.410 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.410 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.410 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.410 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.411 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.411 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.411 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94aeffd640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.411 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94aeffd640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.411 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.411 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94aeffd640 1 -- 172.21.15.155:0/976787639 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94a8011830 con 0x7f94b806b910 2024-08-09T14:50:50.411 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94aeffd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.411 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94aeffd640 1 -- 172.21.15.155:0/976787639 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a8011c20 con 0x7f94b806b910 2024-08-09T14:50:50.411 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.411 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94aeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:50.411 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94aeffd640 10 monclient: dump: 2024-08-09T14:50:50.411 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.412 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.412 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.412 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.412 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.412 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.412 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.412 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.412 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.412 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94bca23640 5 monclient: authenticate success, global_id 4320 2024-08-09T14:50:50.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94bca23640 10 monclient: _renew_subs 2024-08-09T14:50:50.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94bca23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.412 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94bca23640 1 -- 172.21.15.155:0/976787639 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94b8145aa0 con 0x7f94b806b910 2024-08-09T14:50:50.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94b559a640 4 set_mon_vals no callback set 2024-08-09T14:50:50.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94bca23640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94bca23640 10 monclient: _renew_subs 2024-08-09T14:50:50.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94bca23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94bca23640 1 -- 172.21.15.155:0/976787639 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94b81d9240 con 0x7f94b806b910 2024-08-09T14:50:50.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7f94bca23640 1 librados: init done 2024-08-09T14:50:50.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.309+0000 7fac623b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.310+0000 7fac623b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.413 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.310+0000 7fac623b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.310+0000 7f94bca23640 10 client.4320.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.310+0000 7f94bca23640 15 client.4320.objecter _session_command_op_assign -1 1 2024-08-09T14:50:50.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.310+0000 7f94bca23640 20 client.4320.objecter _get_session osd=-1 returning homeless 2024-08-09T14:50:50.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.310+0000 7f94bca23640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.310+0000 7f94bca23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.310+0000 7f94bca23640 1 -- 172.21.15.155:0/976787639 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f94b8072be0 con 0x7f94b806b910 2024-08-09T14:50:50.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.312+0000 7fac623b4640 10 monclient(hunting): handle_auth_done global_id 4323 payload 824 2024-08-09T14:50:50.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.312+0000 7fac623b4640 1 -- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac5c0b0160 msgr2=0x7fac5c14bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.312+0000 7fac623b4640 1 --2- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fac5c0b0160 0x7fac5c14bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.312+0000 7fac623b4640 1 -- 172.21.15.155:0/2011517931 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac5c0b22e0 msgr2=0x7fac5c0aa920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.312+0000 7fac623b4640 1 --2- 172.21.15.155:0/2011517931 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac5c0b22e0 0x7fac5c0aa920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.312+0000 7fac623b4640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.312+0000 7fac623b4640 1 monclient: found mon.noname-a 2024-08-09T14:50:50.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.312+0000 7fac623b4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.312+0000 7fac623b4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.312+0000 7fac623b4640 1 -- 172.21.15.155:0/2011517931 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac5c0aafa0 con 0x7fac5c0b26e0 2024-08-09T14:50:50.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.321+0000 7f6ca77fe640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ca80aa820 0x7f6ca81d26d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43540/0 (socket says 172.21.15.155:43540) 2024-08-09T14:50:50.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.321+0000 7f6ca77fe640 1 -- 172.21.15.155:0/578328842 learned_addr learned my addr 172.21.15.155:0/578328842 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.321+0000 7f6ca77fe640 10 monclient(hunting): get_auth_request con 0x7f6ca80aa820 auth_method 0 2024-08-09T14:50:50.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.321+0000 7f6ca77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.321+0000 7f6ca77fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.321+0000 7f6ca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.321+0000 7fa686ffd640 1 -- 172.21.15.155:0/1179127270 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa69000ac30 con 0x7fa6980aa820 2024-08-09T14:50:50.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.322+0000 7fa686ffd640 1 --2- 172.21.15.155:0/1179127270 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa67806d810 0x7fa67806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.322+0000 7fa695d9b640 1 --2- 172.21.15.155:0/1179127270 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa67806d810 0x7fa67806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.322+0000 7fa695d9b640 10 monclient: get_auth_request con 0x7fa67806d810 auth_method 0 2024-08-09T14:50:50.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.322+0000 7f2472ffd640 1 -- 172.21.15.155:0/1033534714 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f247801a070 con 0x7f247c0aefc0 2024-08-09T14:50:50.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.323+0000 7fa686ffd640 10 client.4282.objecter ms_handle_connect 0x7fa67806d810 2024-08-09T14:50:50.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.323+0000 7fa695d9b640 1 --2- 172.21.15.155:0/1179127270 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa67806d810 0x7fa67806fcd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa6880094a0 tx=0x7fa6880093f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.323+0000 7f2472ffd640 1 --2- 172.21.15.155:0/1033534714 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f246006d810 0x7f246006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.323+0000 7f2481ccc640 1 --2- 172.21.15.155:0/1033534714 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f246006d810 0x7f246006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.323+0000 7f2481ccc640 10 monclient: get_auth_request con 0x7f246006d810 auth_method 0 2024-08-09T14:50:50.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.323+0000 7f2472ffd640 10 client.4274.objecter ms_handle_connect 0x7f246006d810 2024-08-09T14:50:50.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.323+0000 7f2481ccc640 1 --2- 172.21.15.155:0/1033534714 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f246006d810 0x7f246006fcd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f246c038000 tx=0x7f246c03b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.324+0000 7fa686ffd640 1 -- 172.21.15.155:0/1179127270 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7fa690024e70 con 0x7fa6980aa820 2024-08-09T14:50:50.419 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.324+0000 7fa686ffd640 10 client.4282.objecter ms_dispatch 0x7fa698000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:50.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.324+0000 7fa686ffd640 3 client.4282.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:50.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.324+0000 7fa686ffd640 3 client.4282.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:50.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa686ffd640 10 client.4282.objecter checking command 1 2024-08-09T14:50:50.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa686ffd640 1 --2- 172.21.15.155:0/1179127270 >> [v2:172.21.15.155:6802/961143006,v1:172.21.15.155:6803/961143006] conn(0x7fa678073c50 0x7fa678076090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa686ffd640 20 client.4282.objecter _get_session s=0x7fa678073580 osd=2 3 2024-08-09T14:50:50.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa686ffd640 20 client.4282.objecter put_session s=0x7fa678073580 osd=2 3 2024-08-09T14:50:50.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa686ffd640 15 client.4282.objecter _session_command_op_remove -1 1 2024-08-09T14:50:50.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa686ffd640 20 client.4282.objecter _get_session s=0x7fa678073580 osd=2 3 2024-08-09T14:50:50.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa686ffd640 20 client.4282.objecter get_session s=0x7fa678073580 osd=2 3 2024-08-09T14:50:50.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa686ffd640 15 client.4282.objecter _session_command_op_assign 2 1 2024-08-09T14:50:50.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa686ffd640 20 client.4282.objecter put_session s=0x7fa678073580 osd=2 4 2024-08-09T14:50:50.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa686ffd640 10 client.4282.objecter _send_command 1 2024-08-09T14:50:50.420 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa686ffd640 1 -- 172.21.15.155:0/1179127270 --> [v2:172.21.15.155:6802/961143006,v1:172.21.15.155:6803/961143006] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa69000bf30 con 0x7fa678073c50 2024-08-09T14:50:50.421 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa686ffd640 20 client.4282.objecter dump_active .. 0 homeless 2024-08-09T14:50:50.421 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa686ffd640 1 -- 172.21.15.155:0/1179127270 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa690055dc0 con 0x7fa6980aa820 2024-08-09T14:50:50.421 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa686ffd640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-08-09T14:50:50.421 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.325+0000 7fa696d9d640 1 --2- 172.21.15.155:0/1179127270 >> [v2:172.21.15.155:6802/961143006,v1:172.21.15.155:6803/961143006] conn(0x7fa678073c50 0x7fa678076090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.421 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.326+0000 7fa696d9d640 10 monclient: get_auth_request con 0x7fa678073c50 auth_method 0 2024-08-09T14:50:50.421 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.326+0000 7f6ca7fff640 10 monclient(hunting): get_auth_request con 0x7f6ca80ae9c0 auth_method 0 2024-08-09T14:50:50.421 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.326+0000 7f6ca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.326+0000 7f6ca7fff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.326+0000 7f6ca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.327+0000 7f6ca7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.327+0000 7f6ca7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.327+0000 7f6ca7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.327+0000 7f2472ffd640 1 -- 172.21.15.155:0/1033534714 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f2478057370 con 0x7f247c0aefc0 2024-08-09T14:50:50.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.327+0000 7f2472ffd640 10 client.4274.objecter ms_dispatch 0x7f247c000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:50.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.327+0000 7f2472ffd640 3 client.4274.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:50.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.327+0000 7f2472ffd640 3 client.4274.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:50.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2472ffd640 10 client.4274.objecter checking command 1 2024-08-09T14:50:50.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2472ffd640 1 --2- 172.21.15.155:0/1033534714 >> [v2:172.21.15.194:6800/2229949751,v1:172.21.15.194:6801/2229949751] conn(0x7f2460073c50 0x7f2460076090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2472ffd640 20 client.4274.objecter _get_session s=0x7f2460073580 osd=4 3 2024-08-09T14:50:50.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2472ffd640 20 client.4274.objecter put_session s=0x7f2460073580 osd=4 3 2024-08-09T14:50:50.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2472ffd640 15 client.4274.objecter _session_command_op_remove -1 1 2024-08-09T14:50:50.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2472ffd640 20 client.4274.objecter _get_session s=0x7f2460073580 osd=4 3 2024-08-09T14:50:50.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2472ffd640 20 client.4274.objecter get_session s=0x7f2460073580 osd=4 3 2024-08-09T14:50:50.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2472ffd640 15 client.4274.objecter _session_command_op_assign 4 1 2024-08-09T14:50:50.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2472ffd640 20 client.4274.objecter put_session s=0x7f2460073580 osd=4 4 2024-08-09T14:50:50.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2472ffd640 10 client.4274.objecter _send_command 1 2024-08-09T14:50:50.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2472ffd640 1 -- 172.21.15.155:0/1033534714 --> [v2:172.21.15.194:6800/2229949751,v1:172.21.15.194:6801/2229949751] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2478007e50 con 0x7f2460073c50 2024-08-09T14:50:50.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2472ffd640 20 client.4274.objecter dump_active .. 0 homeless 2024-08-09T14:50:50.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2472ffd640 1 -- 172.21.15.155:0/1033534714 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f2478055570 con 0x7f247c0aefc0 2024-08-09T14:50:50.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2472ffd640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-08-09T14:50:50.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.328+0000 7f2482cce640 1 --2- 172.21.15.155:0/1033534714 >> [v2:172.21.15.194:6800/2229949751,v1:172.21.15.194:6801/2229949751] conn(0x7f2460073c50 0x7f2460076090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.329+0000 7f2482cce640 10 monclient: get_auth_request con 0x7f2460073c50 auth_method 0 2024-08-09T14:50:50.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.330+0000 7f6ca7fff640 10 monclient(hunting): handle_auth_done global_id 4294 payload 824 2024-08-09T14:50:50.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.330+0000 7f6ca7fff640 1 -- 172.21.15.155:0/578328842 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ca80aac00 msgr2=0x7f6ca8148330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.330+0000 7fa686ffd640 10 client.4282.objecter ms_handle_connect 0x7fa678073c50 2024-08-09T14:50:50.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.330+0000 7fa696d9d640 1 --2- 172.21.15.155:0/1179127270 >> [v2:172.21.15.155:6802/961143006,v1:172.21.15.155:6803/961143006] conn(0x7fa678073c50 0x7fa678076090 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.330+0000 7f6ca7fff640 1 --2- 172.21.15.155:0/578328842 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6ca80aac00 0x7f6ca8148330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.330+0000 7f6ca7fff640 1 -- 172.21.15.155:0/578328842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ca80aa820 msgr2=0x7f6ca81d26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.330+0000 7f6ca7fff640 1 --2- 172.21.15.155:0/578328842 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6ca80aa820 0x7f6ca81d26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.330+0000 7f6ca7fff640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.330+0000 7f6ca7fff640 1 monclient: found mon.noname-b 2024-08-09T14:50:50.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.330+0000 7f6ca7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.330+0000 7f6ca7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.330+0000 7f6ca7fff640 1 -- 172.21.15.155:0/578328842 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca8148b90 con 0x7f6ca80ae9c0 2024-08-09T14:50:50.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.332+0000 7f2472ffd640 10 client.4274.objecter ms_handle_connect 0x7f2460073c50 2024-08-09T14:50:50.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.332+0000 7f2482cce640 1 --2- 172.21.15.155:0/1033534714 >> [v2:172.21.15.194:6800/2229949751,v1:172.21.15.194:6801/2229949751] conn(0x7f2460073c50 0x7f2460076090 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.334+0000 7f6ca7fff640 1 --2- 172.21.15.155:0/578328842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ca80ae9c0 0x7f6ca81dc420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c00c3c0 tx=0x7f6c9c00c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.334+0000 7f6c8ffff640 10 client.?.objecter ms_handle_connect 0x7f6ca80ae9c0 2024-08-09T14:50:50.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.334+0000 7f6c8ffff640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:50.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.351+0000 7f94aeffd640 1 -- 172.21.15.155:0/976787639 <== mon.0 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f94a8018020 con 0x7f94b806b910 2024-08-09T14:50:50.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.351+0000 7f94aeffd640 1 --2- 172.21.15.155:0/976787639 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f949806d810 0x7f949806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.352+0000 7f94affff640 1 --2- 172.21.15.155:0/976787639 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f949806d810 0x7f949806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.352+0000 7f94affff640 10 monclient: get_auth_request con 0x7f949806d810 auth_method 0 2024-08-09T14:50:50.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.352+0000 7f94affff640 1 --2- 172.21.15.155:0/976787639 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f949806d810 0x7f949806fcd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f94b81450f0 tx=0x7f94a0006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.352+0000 7f94aeffd640 10 client.4320.objecter ms_handle_connect 0x7f949806d810 2024-08-09T14:50:50.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 1 -- 172.21.15.155:0/976787639 <== mon.0 v2:172.21.15.155:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f94a8057760 con 0x7f94b806b910 2024-08-09T14:50:50.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 10 client.4320.objecter ms_dispatch 0x7f94b8000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:50.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 3 client.4320.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:50.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 3 client.4320.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:50.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 10 client.4320.objecter checking command 1 2024-08-09T14:50:50.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 1 --2- 172.21.15.155:0/976787639 >> [v2:172.21.15.194:6820/2670313675,v1:172.21.15.194:6823/2670313675] conn(0x7f9498073c50 0x7f9498076090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 20 client.4320.objecter _get_session s=0x7f9498073580 osd=7 3 2024-08-09T14:50:50.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 20 client.4320.objecter put_session s=0x7f9498073580 osd=7 3 2024-08-09T14:50:50.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 15 client.4320.objecter _session_command_op_remove -1 1 2024-08-09T14:50:50.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 20 client.4320.objecter _get_session s=0x7f9498073580 osd=7 3 2024-08-09T14:50:50.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 20 client.4320.objecter get_session s=0x7f9498073580 osd=7 3 2024-08-09T14:50:50.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 15 client.4320.objecter _session_command_op_assign 7 1 2024-08-09T14:50:50.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 20 client.4320.objecter put_session s=0x7f9498073580 osd=7 4 2024-08-09T14:50:50.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 10 client.4320.objecter _send_command 1 2024-08-09T14:50:50.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 1 -- 172.21.15.155:0/976787639 --> [v2:172.21.15.194:6820/2670313675,v1:172.21.15.194:6823/2670313675] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f94a8006730 con 0x7f9498073c50 2024-08-09T14:50:50.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 20 client.4320.objecter dump_active .. 0 homeless 2024-08-09T14:50:50.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 1 -- 172.21.15.155:0/976787639 <== mon.0 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f94a8055960 con 0x7f94b806b910 2024-08-09T14:50:50.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.357+0000 7f94aeffd640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-08-09T14:50:50.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.358+0000 7f94b6d9d640 1 --2- 172.21.15.155:0/976787639 >> [v2:172.21.15.194:6820/2670313675,v1:172.21.15.194:6823/2670313675] conn(0x7f9498073c50 0x7f9498076090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.358+0000 7f94b6d9d640 10 monclient: get_auth_request con 0x7f9498073c50 auth_method 0 2024-08-09T14:50:50.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.361+0000 7f94b6d9d640 1 --2- 172.21.15.155:0/976787639 >> [v2:172.21.15.194:6820/2670313675,v1:172.21.15.194:6823/2670313675] conn(0x7f9498073c50 0x7f9498076090 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.431 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.361+0000 7f94aeffd640 10 client.4320.objecter ms_handle_connect 0x7f9498073c50 2024-08-09T14:50:50.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.361+0000 7f0be5ffb640 1 -- 172.21.15.155:0/2103563403 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0bec017070 con 0x7f0bf0071010 2024-08-09T14:50:50.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.362+0000 7f0be5ffb640 1 --2- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f0bd406d7c0 0x7f0bd406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.362+0000 7f0bf4e7c640 1 --2- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f0bd406d7c0 0x7f0bd406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.362+0000 7f0bf4e7c640 10 monclient: get_auth_request con 0x7f0bd406d7c0 auth_method 0 2024-08-09T14:50:50.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.362+0000 7f0bf4e7c640 1 --2- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f0bd406d7c0 0x7f0bd406fc80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0be0036000 tx=0x7f0be003d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.362+0000 7f0be5ffb640 10 client.4277.objecter ms_handle_connect 0x7f0bd406d7c0 2024-08-09T14:50:50.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.366+0000 7f3d127fc640 1 -- 172.21.15.155:0/4090729798 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3d1804b060 con 0x7f3d140c34e0 2024-08-09T14:50:50.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.366+0000 7f0be5ffb640 1 -- 172.21.15.155:0/2103563403 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f0bec0571f0 con 0x7f0bf0071010 2024-08-09T14:50:50.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.366+0000 7f0be5ffb640 10 client.4277.objecter ms_dispatch 0x7f0bf0000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:50.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.366+0000 7f0be5ffb640 3 client.4277.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:50.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.366+0000 7f0be5ffb640 3 client.4277.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:50.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.366+0000 7f3d127fc640 1 --2- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f3d0006d840 0x7f3d0006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.366+0000 7f0be5ffb640 10 client.4277.objecter checking command 1 2024-08-09T14:50:50.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.366+0000 7f0be5ffb640 1 --2- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:6818/3129678288,v1:172.21.15.155:6819/3129678288] conn(0x7f0bd4073c00 0x7f0bd4076040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.366+0000 7f0be5ffb640 20 client.4277.objecter _get_session s=0x7f0bd4073530 osd=1 3 2024-08-09T14:50:50.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f0be5ffb640 20 client.4277.objecter put_session s=0x7f0bd4073530 osd=1 3 2024-08-09T14:50:50.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f0be5ffb640 15 client.4277.objecter _session_command_op_remove -1 1 2024-08-09T14:50:50.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f0be5ffb640 20 client.4277.objecter _get_session s=0x7f0bd4073530 osd=1 3 2024-08-09T14:50:50.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f0be5ffb640 20 client.4277.objecter get_session s=0x7f0bd4073530 osd=1 3 2024-08-09T14:50:50.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f0be5ffb640 15 client.4277.objecter _session_command_op_assign 1 1 2024-08-09T14:50:50.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f0be5ffb640 20 client.4277.objecter put_session s=0x7f0bd4073530 osd=1 4 2024-08-09T14:50:50.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f0be5ffb640 10 client.4277.objecter _send_command 1 2024-08-09T14:50:50.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f0be5ffb640 1 -- 172.21.15.155:0/2103563403 --> [v2:172.21.15.155:6818/3129678288,v1:172.21.15.155:6819/3129678288] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0bd4076710 con 0x7f0bd4073c00 2024-08-09T14:50:50.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f0be5ffb640 20 client.4277.objecter dump_active .. 0 homeless 2024-08-09T14:50:50.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f0be5ffb640 1 -- 172.21.15.155:0/2103563403 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0bec0553f0 con 0x7f0bf0071010 2024-08-09T14:50:50.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f0be5ffb640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-08-09T14:50:50.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f3d21c20640 1 --2- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f3d0006d840 0x7f3d0006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f3d21c20640 10 monclient: get_auth_request con 0x7f3d0006d840 auth_method 0 2024-08-09T14:50:50.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f3d127fc640 10 client.4288.objecter ms_handle_connect 0x7f3d0006d840 2024-08-09T14:50:50.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f3d21c20640 1 --2- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f3d0006d840 0x7f3d0006fd00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c050f50 tx=0x7f3d1c0675c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.367+0000 7f0bf5e7e640 1 --2- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:6818/3129678288,v1:172.21.15.155:6819/3129678288] conn(0x7f0bd4073c00 0x7f0bd4076040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.368+0000 7f0bf5e7e640 10 monclient: get_auth_request con 0x7f0bd4073c00 auth_method 0 2024-08-09T14:50:50.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.372+0000 7f0bf5e7e640 1 --2- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:6818/3129678288,v1:172.21.15.155:6819/3129678288] conn(0x7f0bd4073c00 0x7f0bd4076040 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.372+0000 7f0be5ffb640 10 client.4277.objecter ms_handle_connect 0x7f0bd4073c00 2024-08-09T14:50:50.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.373+0000 7f20157fa640 1 -- 172.21.15.155:0/4032070454 <== mon.0 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f200c018020 con 0x7f20180b0160 2024-08-09T14:50:50.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.374+0000 7f20157fa640 1 --2- 172.21.15.155:0/4032070454 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f1ffc06d810 0x7f1ffc06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.374+0000 7f2017fff640 1 --2- 172.21.15.155:0/4032070454 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f1ffc06d810 0x7f1ffc06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.374+0000 7f2017fff640 10 monclient: get_auth_request con 0x7f1ffc06d810 auth_method 0 2024-08-09T14:50:50.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.374+0000 7f20157fa640 10 client.4317.objecter ms_handle_connect 0x7f1ffc06d810 2024-08-09T14:50:50.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.374+0000 7f2017fff640 1 --2- 172.21.15.155:0/4032070454 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f1ffc06d810 0x7f1ffc06fcd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f20180aab00 tx=0x7f2004009110 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.438 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 1 -- 172.21.15.155:0/4090729798 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f3d180793a0 con 0x7f3d140c34e0 2024-08-09T14:50:50.438 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 10 client.4288.objecter ms_dispatch 0x7f3d1c000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:50.438 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 3 client.4288.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:50.438 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 3 client.4288.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:50.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 10 client.4288.objecter checking command 1 2024-08-09T14:50:50.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 1 --2- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] conn(0x7f3d00073c80 0x7f3d000760c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 20 client.4288.objecter _get_session s=0x7f3d000735b0 osd=0 3 2024-08-09T14:50:50.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 20 client.4288.objecter put_session s=0x7f3d000735b0 osd=0 3 2024-08-09T14:50:50.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 15 client.4288.objecter _session_command_op_remove -1 1 2024-08-09T14:50:50.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 20 client.4288.objecter _get_session s=0x7f3d000735b0 osd=0 3 2024-08-09T14:50:50.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 20 client.4288.objecter get_session s=0x7f3d000735b0 osd=0 3 2024-08-09T14:50:50.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 15 client.4288.objecter _session_command_op_assign 0 1 2024-08-09T14:50:50.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 20 client.4288.objecter put_session s=0x7f3d000735b0 osd=0 4 2024-08-09T14:50:50.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 10 client.4288.objecter _send_command 1 2024-08-09T14:50:50.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 1 -- 172.21.15.155:0/4090729798 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d18008f30 con 0x7f3d00073c80 2024-08-09T14:50:50.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.376+0000 7f3d127fc640 20 client.4288.objecter dump_active .. 0 homeless 2024-08-09T14:50:50.439 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.377+0000 7f3d127fc640 1 -- 172.21.15.155:0/4090729798 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3d180775a0 con 0x7f3d140c34e0 2024-08-09T14:50:50.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.377+0000 7f3d127fc640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-08-09T14:50:50.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.377+0000 7f3d22421640 1 --2- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] conn(0x7f3d00073c80 0x7f3d000760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.377+0000 7f3d22421640 10 monclient: get_auth_request con 0x7f3d00073c80 auth_method 0 2024-08-09T14:50:50.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.379+0000 7fac623b4640 1 --2- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac5c0b26e0 0x7fac5c1495d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fac5400efc0 tx=0x7fac5400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.440 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.379+0000 7fac527fc640 10 client.?.objecter ms_handle_connect 0x7fac5c0b26e0 2024-08-09T14:50:50.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6c8ffff640 1 -- 172.21.15.155:0/578328842 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c017070 con 0x7f6ca80ae9c0 2024-08-09T14:50:50.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6c8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6c8ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:50.441 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6c8ffff640 10 monclient: dump: 2024-08-09T14:50:50.441 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.441 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.441 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.441 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.441 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.441 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.441 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.441 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.441 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.442 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.442 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6c8ffff640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.442 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6c8ffff640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.442 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6c8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.442 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6c8ffff640 1 -- 172.21.15.155:0/578328842 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c9c005ce0 con 0x7f6ca80ae9c0 2024-08-09T14:50:50.442 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6c8ffff640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.442 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6c8ffff640 1 -- 172.21.15.155:0/578328842 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c0072f0 con 0x7f6ca80ae9c0 2024-08-09T14:50:50.442 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6c8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.442 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6c8ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:50.442 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6c8ffff640 10 monclient: dump: 2024-08-09T14:50:50.442 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.442 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.442 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.443 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.443 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.443 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.443 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.443 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.443 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.443 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.443 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.379+0000 7fac527fc640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:50.444 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6cae0a0640 5 monclient: authenticate success, global_id 4294 2024-08-09T14:50:50.444 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6cae0a0640 10 monclient: _renew_subs 2024-08-09T14:50:50.444 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6cae0a0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.444 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6cae0a0640 1 -- 172.21.15.155:0/578328842 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca81488d0 con 0x7f6ca80ae9c0 2024-08-09T14:50:50.444 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6ca5ffb640 4 set_mon_vals no callback set 2024-08-09T14:50:50.444 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6cae0a0640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.444 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6cae0a0640 10 monclient: _renew_subs 2024-08-09T14:50:50.444 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6cae0a0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.444 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6cae0a0640 1 -- 172.21.15.155:0/578328842 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca81dc960 con 0x7f6ca80ae9c0 2024-08-09T14:50:50.444 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f6cae0a0640 1 librados: init done 2024-08-09T14:50:50.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 1 -- 172.21.15.155:0/4032070454 <== mon.0 v2:172.21.15.155:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f200c057700 con 0x7f20180b0160 2024-08-09T14:50:50.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 10 client.4317.objecter ms_dispatch 0x7f2018000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:50.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 3 client.4317.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:50.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 3 client.4317.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:50.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 10 client.4317.objecter checking command 1 2024-08-09T14:50:50.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 1 --2- 172.21.15.155:0/4032070454 >> [v2:172.21.15.194:6808/3077539270,v1:172.21.15.194:6809/3077539270] conn(0x7f1ffc073c50 0x7f1ffc076090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 20 client.4317.objecter _get_session s=0x7f1ffc073580 osd=6 3 2024-08-09T14:50:50.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 20 client.4317.objecter put_session s=0x7f1ffc073580 osd=6 3 2024-08-09T14:50:50.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 15 client.4317.objecter _session_command_op_remove -1 1 2024-08-09T14:50:50.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 20 client.4317.objecter _get_session s=0x7f1ffc073580 osd=6 3 2024-08-09T14:50:50.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 20 client.4317.objecter get_session s=0x7f1ffc073580 osd=6 3 2024-08-09T14:50:50.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 15 client.4317.objecter _session_command_op_assign 6 1 2024-08-09T14:50:50.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 20 client.4317.objecter put_session s=0x7f1ffc073580 osd=6 4 2024-08-09T14:50:50.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 10 client.4317.objecter _send_command 1 2024-08-09T14:50:50.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 1 -- 172.21.15.155:0/4032070454 --> [v2:172.21.15.194:6808/3077539270,v1:172.21.15.194:6809/3077539270] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f200c006820 con 0x7f1ffc073c50 2024-08-09T14:50:50.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 20 client.4317.objecter dump_active .. 0 homeless 2024-08-09T14:50:50.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 1 -- 172.21.15.155:0/4032070454 <== mon.0 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f200c055900 con 0x7f20180b0160 2024-08-09T14:50:50.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.380+0000 7f20157fa640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-08-09T14:50:50.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.381+0000 7f6cae0a0640 10 client.4294.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.381+0000 7f6cae0a0640 15 client.4294.objecter _session_command_op_assign -1 1 2024-08-09T14:50:50.447 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.381+0000 7f6cae0a0640 20 client.4294.objecter _get_session osd=-1 returning homeless 2024-08-09T14:50:50.447 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.381+0000 7f6cae0a0640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.447 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.381+0000 7f6cae0a0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.447 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.381+0000 7f6cae0a0640 1 -- 172.21.15.155:0/578328842 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6ca8144290 con 0x7f6ca80ae9c0 2024-08-09T14:50:50.447 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.381+0000 7f201d38c640 1 --2- 172.21.15.155:0/4032070454 >> [v2:172.21.15.194:6808/3077539270,v1:172.21.15.194:6809/3077539270] conn(0x7f1ffc073c50 0x7f1ffc076090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.447 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.382+0000 7f201d38c640 10 monclient: get_auth_request con 0x7f1ffc073c50 auth_method 0 2024-08-09T14:50:50.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.382+0000 7f3d127fc640 10 client.4288.objecter ms_handle_connect 0x7f3d00073c80 2024-08-09T14:50:50.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.382+0000 7f3d22421640 1 --2- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] conn(0x7f3d00073c80 0x7f3d000760c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.385+0000 7f201d38c640 1 --2- 172.21.15.155:0/4032070454 >> [v2:172.21.15.194:6808/3077539270,v1:172.21.15.194:6809/3077539270] conn(0x7f1ffc073c50 0x7f1ffc076090 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.385+0000 7f20157fa640 10 client.4317.objecter ms_handle_connect 0x7f1ffc073c50 2024-08-09T14:50:50.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.391+0000 7fac527fc640 1 -- 172.21.15.155:0/2011517931 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac54018020 con 0x7fac5c0b26e0 2024-08-09T14:50:50.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.391+0000 7fac527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.391+0000 7fac527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:50.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.391+0000 7fac527fc640 10 monclient: dump: 2024-08-09T14:50:50.449 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.449 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.449 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.449 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.450 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.450 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.450 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.450 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.450 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.450 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.391+0000 7fac527fc640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.391+0000 7fac527fc640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.391+0000 7fac527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.391+0000 7fac527fc640 1 -- 172.21.15.155:0/2011517931 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac54014480 con 0x7fac5c0b26e0 2024-08-09T14:50:50.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.391+0000 7fac527fc640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.450 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.391+0000 7fac527fc640 1 -- 172.21.15.155:0/2011517931 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac54014870 con 0x7fac5c0b26e0 2024-08-09T14:50:50.451 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.391+0000 7fac527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.451 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.391+0000 7fac527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:50.451 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.391+0000 7fac527fc640 10 monclient: dump: 2024-08-09T14:50:50.451 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.451 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.451 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.451 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.451 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.451 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.451 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.451 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.451 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.451 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.451 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.391+0000 7fac53fff640 4 set_mon_vals no callback set 2024-08-09T14:50:50.452 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.392+0000 7fac63e17640 5 monclient: authenticate success, global_id 4323 2024-08-09T14:50:50.452 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.392+0000 7fac63e17640 10 monclient: _renew_subs 2024-08-09T14:50:50.452 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.392+0000 7fac63e17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.452 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.392+0000 7fac63e17640 1 -- 172.21.15.155:0/2011517931 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac5c0aba40 con 0x7fac5c0b26e0 2024-08-09T14:50:50.452 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.392+0000 7fac63e17640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.452 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.392+0000 7fac63e17640 10 monclient: _renew_subs 2024-08-09T14:50:50.452 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.392+0000 7fac63e17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.452 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.392+0000 7fac63e17640 1 -- 172.21.15.155:0/2011517931 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac5c0ac000 con 0x7fac5c0b26e0 2024-08-09T14:50:50.452 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.392+0000 7fac63e17640 1 librados: init done 2024-08-09T14:50:50.452 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.393+0000 7fac63e17640 10 client.4323.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.452 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.393+0000 7fac63e17640 15 client.4323.objecter _session_command_op_assign -1 1 2024-08-09T14:50:50.452 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.393+0000 7fac63e17640 20 client.4323.objecter _get_session osd=-1 returning homeless 2024-08-09T14:50:50.452 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.393+0000 7fac63e17640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.393+0000 7fac63e17640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.393+0000 7fac63e17640 1 -- 172.21.15.155:0/2011517931 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fac5c0b0540 con 0x7fac5c0b26e0 2024-08-09T14:50:50.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.397+0000 7f2472ffd640 1 -- 172.21.15.155:0/1033534714 <== osd.4 v2:172.21.15.194:6800/2229949751 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2478007e50 con 0x7f2460073c50 2024-08-09T14:50:50.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.397+0000 7f2472ffd640 10 client.4274.objecter ms_dispatch 0x7f247c000b90 command_reply(tid 1: 0 ) 2024-08-09T14:50:50.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.397+0000 7f2472ffd640 10 client.4274.objecter _finish_command 1 = system:0 2024-08-09T14:50:50.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.397+0000 7f2472ffd640 20 client.4274.objecter put_session s=0x7f2460073580 osd=4 4 2024-08-09T14:50:50.453 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.397+0000 7f2472ffd640 15 client.4274.objecter _session_command_op_remove 4 1 2024-08-09T14:50:50.454 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.410+0000 7fa686ffd640 1 -- 172.21.15.155:0/1179127270 <== osd.2 v2:172.21.15.155:6802/961143006 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa69000bf30 con 0x7fa678073c50 2024-08-09T14:50:50.454 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.410+0000 7fa686ffd640 10 client.4282.objecter ms_dispatch 0x7fa698000b90 command_reply(tid 1: 0 ) 2024-08-09T14:50:50.454 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.410+0000 7fa686ffd640 10 client.4282.objecter _finish_command 1 = system:0 2024-08-09T14:50:50.454 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.410+0000 7fa686ffd640 20 client.4282.objecter put_session s=0x7fa678073580 osd=2 4 2024-08-09T14:50:50.454 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.410+0000 7fa686ffd640 15 client.4282.objecter _session_command_op_remove 2 1 2024-08-09T14:50:50.454 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.414+0000 7f6c8ffff640 1 -- 172.21.15.155:0/578328842 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6c9c01a070 con 0x7f6ca80ae9c0 2024-08-09T14:50:50.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.415+0000 7f6c8ffff640 1 --2- 172.21.15.155:0/578328842 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f6c9406d7c0 0x7f6c9406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.415+0000 7f6ca77fe640 1 --2- 172.21.15.155:0/578328842 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f6c9406d7c0 0x7f6c9406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.415+0000 7f6ca77fe640 10 monclient: get_auth_request con 0x7f6c9406d7c0 auth_method 0 2024-08-09T14:50:50.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.416+0000 7f6ca77fe640 1 --2- 172.21.15.155:0/578328842 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f6c9406d7c0 0x7f6c9406fc80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0000f80 tx=0x7f6ca0030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.416+0000 7f6c8ffff640 10 client.4294.objecter ms_handle_connect 0x7f6c9406d7c0 2024-08-09T14:50:50.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.420+0000 7f94aeffd640 1 -- 172.21.15.155:0/976787639 <== osd.7 v2:172.21.15.194:6820/2670313675 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f94a8006730 con 0x7f9498073c50 2024-08-09T14:50:50.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.420+0000 7f94aeffd640 10 client.4320.objecter ms_dispatch 0x7f94b8000b90 command_reply(tid 1: 0 ) 2024-08-09T14:50:50.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.420+0000 7f94aeffd640 10 client.4320.objecter _finish_command 1 = system:0 2024-08-09T14:50:50.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.420+0000 7f94aeffd640 20 client.4320.objecter put_session s=0x7f9498073580 osd=7 4 2024-08-09T14:50:50.455 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.420+0000 7f94aeffd640 15 client.4320.objecter _session_command_op_remove 7 1 2024-08-09T14:50:50.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 1 -- 172.21.15.155:0/578328842 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f6c9c056cd0 con 0x7f6ca80ae9c0 2024-08-09T14:50:50.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 10 client.4294.objecter ms_dispatch 0x7f6ca8000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:50.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 3 client.4294.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:50.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 3 client.4294.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:50.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 10 client.4294.objecter checking command 1 2024-08-09T14:50:50.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 1 --2- 172.21.15.155:0/578328842 >> [v2:172.21.15.194:6816/1013364343,v1:172.21.15.194:6817/1013364343] conn(0x7f6c940745c0 0x7f6c94076a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 20 client.4294.objecter _get_session s=0x7f6c94073ef0 osd=5 3 2024-08-09T14:50:50.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 20 client.4294.objecter put_session s=0x7f6c94073ef0 osd=5 3 2024-08-09T14:50:50.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 15 client.4294.objecter _session_command_op_remove -1 1 2024-08-09T14:50:50.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 20 client.4294.objecter _get_session s=0x7f6c94073ef0 osd=5 3 2024-08-09T14:50:50.457 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 20 client.4294.objecter get_session s=0x7f6c94073ef0 osd=5 3 2024-08-09T14:50:50.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 15 client.4294.objecter _session_command_op_assign 5 1 2024-08-09T14:50:50.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 20 client.4294.objecter put_session s=0x7f6c94073ef0 osd=5 4 2024-08-09T14:50:50.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 10 client.4294.objecter _send_command 1 2024-08-09T14:50:50.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 1 -- 172.21.15.155:0/578328842 --> [v2:172.21.15.194:6816/1013364343,v1:172.21.15.194:6817/1013364343] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c94077130 con 0x7f6c940745c0 2024-08-09T14:50:50.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 20 client.4294.objecter dump_active .. 0 homeless 2024-08-09T14:50:50.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 1 -- 172.21.15.155:0/578328842 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6c9c055660 con 0x7f6ca80ae9c0 2024-08-09T14:50:50.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6c8ffff640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-08-09T14:50:50.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.424+0000 7f6ca6ffd640 1 --2- 172.21.15.155:0/578328842 >> [v2:172.21.15.194:6816/1013364343,v1:172.21.15.194:6817/1013364343] conn(0x7f6c940745c0 0x7f6c94076a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.425+0000 7f6ca6ffd640 10 monclient: get_auth_request con 0x7f6c940745c0 auth_method 0 2024-08-09T14:50:50.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.425+0000 7fa69cbed640 10 client.4282.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-09T14:50:50.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.425+0000 7fa69cbed640 15 client.4282.objecter _session_command_op_assign -1 2 2024-08-09T14:50:50.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.425+0000 7fa69cbed640 20 client.4282.objecter _get_session s=0x7fa678073580 osd=2 3 2024-08-09T14:50:50.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.425+0000 7fa69cbed640 20 client.4282.objecter put_session s=0x7fa678073580 osd=2 3 2024-08-09T14:50:50.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.425+0000 7fa69cbed640 20 client.4282.objecter _get_session s=0x7fa678073580 osd=2 3 2024-08-09T14:50:50.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.425+0000 7fa69cbed640 15 client.4282.objecter _session_command_op_remove -1 2 2024-08-09T14:50:50.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.425+0000 7fa69cbed640 20 client.4282.objecter get_session s=0x7fa678073580 osd=2 3 2024-08-09T14:50:50.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.425+0000 7fa69cbed640 15 client.4282.objecter _session_command_op_assign 2 2 2024-08-09T14:50:50.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.425+0000 7fa69cbed640 20 client.4282.objecter put_session s=0x7fa678073580 osd=2 4 2024-08-09T14:50:50.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.425+0000 7fa69cbed640 10 client.4282.objecter _send_command 2 2024-08-09T14:50:50.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.425+0000 7fa69cbed640 1 -- 172.21.15.155:0/1179127270 --> [v2:172.21.15.155:6802/961143006,v1:172.21.15.155:6803/961143006] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa698144290 con 0x7fa678073c50 2024-08-09T14:50:50.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.428+0000 7f2484731640 10 client.4274.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-09T14:50:50.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.428+0000 7f2484731640 15 client.4274.objecter _session_command_op_assign -1 2 2024-08-09T14:50:50.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.428+0000 7f2484731640 20 client.4274.objecter _get_session s=0x7f2460073580 osd=4 3 2024-08-09T14:50:50.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.428+0000 7f2484731640 20 client.4274.objecter put_session s=0x7f2460073580 osd=4 3 2024-08-09T14:50:50.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.428+0000 7f2484731640 20 client.4274.objecter _get_session s=0x7f2460073580 osd=4 3 2024-08-09T14:50:50.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.428+0000 7f2484731640 15 client.4274.objecter _session_command_op_remove -1 2 2024-08-09T14:50:50.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.428+0000 7f2484731640 20 client.4274.objecter get_session s=0x7f2460073580 osd=4 3 2024-08-09T14:50:50.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.428+0000 7f2484731640 15 client.4274.objecter _session_command_op_assign 4 2 2024-08-09T14:50:50.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.428+0000 7f2484731640 20 client.4274.objecter put_session s=0x7f2460073580 osd=4 4 2024-08-09T14:50:50.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.428+0000 7f2484731640 10 client.4274.objecter _send_command 2 2024-08-09T14:50:50.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.428+0000 7f2484731640 1 -- 172.21.15.155:0/1033534714 --> [v2:172.21.15.194:6800/2229949751,v1:172.21.15.194:6801/2229949751] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f247c0aba00 con 0x7f2460073c50 2024-08-09T14:50:50.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.428+0000 7fac527fc640 1 -- 172.21.15.155:0/2011517931 <== mon.0 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fac54005d10 con 0x7fac5c0b26e0 2024-08-09T14:50:50.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.429+0000 7fac527fc640 1 --2- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fac4006d810 0x7fac4006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.429+0000 7f6ca6ffd640 1 --2- 172.21.15.155:0/578328842 >> [v2:172.21.15.194:6816/1013364343,v1:172.21.15.194:6817/1013364343] conn(0x7f6c940745c0 0x7f6c94076a60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.429+0000 7f6c8ffff640 10 client.4294.objecter ms_handle_connect 0x7f6c940745c0 2024-08-09T14:50:50.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.429+0000 7f0be5ffb640 1 -- 172.21.15.155:0/2103563403 <== osd.1 v2:172.21.15.155:6818/3129678288 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0bd4076710 con 0x7f0bd4073c00 2024-08-09T14:50:50.463 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.429+0000 7f0be5ffb640 10 client.4277.objecter ms_dispatch 0x7f0bf0000b90 command_reply(tid 1: 0 ) 2024-08-09T14:50:50.463 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.429+0000 7f0be5ffb640 10 client.4277.objecter _finish_command 1 = system:0 2024-08-09T14:50:50.463 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.429+0000 7f0be5ffb640 20 client.4277.objecter put_session s=0x7f0bd4073530 osd=1 4 2024-08-09T14:50:50.463 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.429+0000 7f0be5ffb640 15 client.4277.objecter _session_command_op_remove 1 1 2024-08-09T14:50:50.463 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.429+0000 7fac61bb3640 1 --2- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fac4006d810 0x7fac4006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.463 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.429+0000 7fac61bb3640 10 monclient: get_auth_request con 0x7fac4006d810 auth_method 0 2024-08-09T14:50:50.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.430+0000 7fac527fc640 10 client.4323.objecter ms_handle_connect 0x7fac4006d810 2024-08-09T14:50:50.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.430+0000 7fac61bb3640 1 --2- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fac4006d810 0x7fac4006fcd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fac580090b0 tx=0x7fac5803d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.435+0000 7f2472ffd640 1 -- 172.21.15.155:0/1033534714 <== osd.4 v2:172.21.15.194:6800/2229949751 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f247c0aba00 con 0x7f2460073c50 2024-08-09T14:50:50.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.435+0000 7f2472ffd640 10 client.4274.objecter ms_dispatch 0x7f247c000b90 command_reply(tid 2: 0 ) 2024-08-09T14:50:50.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.435+0000 7f2472ffd640 10 client.4274.objecter _finish_command 2 = system:0 2024-08-09T14:50:50.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.435+0000 7f2472ffd640 20 client.4274.objecter put_session s=0x7f2460073580 osd=4 4 2024-08-09T14:50:50.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.435+0000 7f2472ffd640 15 client.4274.objecter _session_command_op_remove 4 2 2024-08-09T14:50:50.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.435+0000 7fac527fc640 1 -- 172.21.15.155:0/2011517931 <== mon.0 v2:172.21.15.155:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7fac54058860 con 0x7fac5c0b26e0 2024-08-09T14:50:50.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.435+0000 7fac527fc640 10 client.4323.objecter ms_dispatch 0x7fac5c000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:50.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.435+0000 7fac527fc640 3 client.4323.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:50.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.435+0000 7fac527fc640 3 client.4323.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:50.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7fac527fc640 10 client.4323.objecter checking command 1 2024-08-09T14:50:50.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7fac527fc640 1 --2- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] conn(0x7fac40073c50 0x7fac40076090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7fac527fc640 20 client.4323.objecter _get_session s=0x7fac40073580 osd=3 3 2024-08-09T14:50:50.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7fac527fc640 20 client.4323.objecter put_session s=0x7fac40073580 osd=3 3 2024-08-09T14:50:50.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7fac527fc640 15 client.4323.objecter _session_command_op_remove -1 1 2024-08-09T14:50:50.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7fac527fc640 20 client.4323.objecter _get_session s=0x7fac40073580 osd=3 3 2024-08-09T14:50:50.465 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7fac527fc640 20 client.4323.objecter get_session s=0x7fac40073580 osd=3 3 2024-08-09T14:50:50.466 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7fac527fc640 15 client.4323.objecter _session_command_op_assign 3 1 2024-08-09T14:50:50.466 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7fac527fc640 20 client.4323.objecter put_session s=0x7fac40073580 osd=3 4 2024-08-09T14:50:50.466 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7fac527fc640 10 client.4323.objecter _send_command 1 2024-08-09T14:50:50.466 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7fac527fc640 1 -- 172.21.15.155:0/2011517931 --> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac54004810 con 0x7fac40073c50 2024-08-09T14:50:50.466 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7fac527fc640 20 client.4323.objecter dump_active .. 0 homeless 2024-08-09T14:50:50.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7f2484731640 10 librados: watch_flush enter 2024-08-09T14:50:50.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7f2484731640 10 librados: watch_flush exit 2024-08-09T14:50:50.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7f2484731640 10 client.4274.objecter close_session for osd.4 2024-08-09T14:50:50.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7f2484731640 1 -- 172.21.15.155:0/1033534714 >> [v2:172.21.15.194:6800/2229949751,v1:172.21.15.194:6801/2229949751] conn(0x7f2460073c50 msgr2=0x7f2460076090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7f2484731640 1 --2- 172.21.15.155:0/1033534714 >> [v2:172.21.15.194:6800/2229949751,v1:172.21.15.194:6801/2229949751] conn(0x7f2460073c50 0x7f2460076090 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7f2484731640 20 client.4274.objecter put_session s=0x7f2460073580 osd=4 1 2024-08-09T14:50:50.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7f2484731640 20 client.4274.objecter shutdown clearing up homeless session... 2024-08-09T14:50:50.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.436+0000 7f2484731640 10 client.4274.objecter successfully canceled tick 2024-08-09T14:50:50.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7f2484731640 1 -- 172.21.15.155:0/1033534714 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f246006d810 msgr2=0x7f246006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7f2484731640 1 --2- 172.21.15.155:0/1033534714 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f246006d810 0x7f246006fcd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f246c038000 tx=0x7f246c03b040 comp rx=0 tx=0).stop 2024-08-09T14:50:50.468 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7f2484731640 10 monclient: shutdown 2024-08-09T14:50:50.469 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7f2484731640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.469 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7f2484731640 1 -- 172.21.15.155:0/1033534714 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f247c0aefc0 msgr2=0x7f247c072f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.469 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7f2484731640 1 --2- 172.21.15.155:0/1033534714 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f247c0aefc0 0x7f247c072f40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f247800e8a0 tx=0x7f247800ed60 comp rx=0 tx=0).stop 2024-08-09T14:50:50.469 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7f2482cce640 1 -- 172.21.15.155:0/1033534714 reap_dead start 2024-08-09T14:50:50.469 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7f2484731640 1 -- 172.21.15.155:0/1033534714 shutdown_connections 2024-08-09T14:50:50.469 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7f2484731640 1 -- 172.21.15.155:0/1033534714 >> 172.21.15.155:0/1033534714 conn(0x7f247c06c200 msgr2=0x7f247c0ad110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.469 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7fa686ffd640 1 -- 172.21.15.155:0/1179127270 <== osd.2 v2:172.21.15.155:6802/961143006 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa698144290 con 0x7fa678073c50 2024-08-09T14:50:50.469 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7fa686ffd640 10 client.4282.objecter ms_dispatch 0x7fa698000b90 command_reply(tid 2: 0 ) 2024-08-09T14:50:50.469 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7fa686ffd640 10 client.4282.objecter _finish_command 2 = system:0 2024-08-09T14:50:50.469 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7fa686ffd640 20 client.4282.objecter put_session s=0x7fa678073580 osd=2 4 2024-08-09T14:50:50.469 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7fa686ffd640 15 client.4282.objecter _session_command_op_remove 2 2 2024-08-09T14:50:50.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7f2484731640 1 -- 172.21.15.155:0/1033534714 shutdown_connections 2024-08-09T14:50:50.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7f2484731640 1 -- 172.21.15.155:0/1033534714 wait complete. 2024-08-09T14:50:50.470 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.437+0000 7f2484731640 1 librados: shutdown 2024-08-09T14:50:50.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.438+0000 7fac613b2640 1 --2- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] conn(0x7fac40073c50 0x7fac40076090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.438+0000 7fac613b2640 10 monclient: get_auth_request con 0x7fac40073c50 auth_method 0 2024-08-09T14:50:50.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.438+0000 7fac527fc640 1 -- 172.21.15.155:0/2011517931 <== mon.0 v2:172.21.15.155:3300/0 6 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fac54056a60 con 0x7fac5c0b26e0 2024-08-09T14:50:50.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.438+0000 7fac527fc640 10 monclient: handle_get_version_reply finishing 1 version 27 2024-08-09T14:50:50.471 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.439+0000 7fa69cbed640 10 librados: watch_flush enter 2024-08-09T14:50:50.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.439+0000 7fa69cbed640 10 librados: watch_flush exit 2024-08-09T14:50:50.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.439+0000 7fa69cbed640 10 client.4282.objecter close_session for osd.2 2024-08-09T14:50:50.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.439+0000 7fa69cbed640 1 -- 172.21.15.155:0/1179127270 >> [v2:172.21.15.155:6802/961143006,v1:172.21.15.155:6803/961143006] conn(0x7fa678073c50 msgr2=0x7fa678076090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.439+0000 7fa69cbed640 1 --2- 172.21.15.155:0/1179127270 >> [v2:172.21.15.155:6802/961143006,v1:172.21.15.155:6803/961143006] conn(0x7fa678073c50 0x7fa678076090 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.439+0000 7fa69cbed640 20 client.4282.objecter put_session s=0x7fa678073580 osd=2 1 2024-08-09T14:50:50.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.439+0000 7fa69cbed640 20 client.4282.objecter shutdown clearing up homeless session... 2024-08-09T14:50:50.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.439+0000 7fa69cbed640 10 client.4282.objecter successfully canceled tick 2024-08-09T14:50:50.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.440+0000 7fa69cbed640 1 -- 172.21.15.155:0/1179127270 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa67806d810 msgr2=0x7fa67806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.440+0000 7fa69cbed640 1 --2- 172.21.15.155:0/1179127270 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa67806d810 0x7fa67806fcd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa6880094a0 tx=0x7fa6880093f0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.440+0000 7fa69cbed640 10 monclient: shutdown 2024-08-09T14:50:50.472 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.440+0000 7fa69cbed640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.440+0000 7fa69cbed640 1 -- 172.21.15.155:0/1179127270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6980aa820 msgr2=0x7fa698134bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.440+0000 7fa69cbed640 1 --2- 172.21.15.155:0/1179127270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6980aa820 0x7fa698134bd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa69000c870 tx=0x7fa69000cd30 comp rx=0 tx=0).stop 2024-08-09T14:50:50.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.440+0000 7fa696d9d640 1 -- 172.21.15.155:0/1179127270 reap_dead start 2024-08-09T14:50:50.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.440+0000 7fa69cbed640 1 -- 172.21.15.155:0/1179127270 shutdown_connections 2024-08-09T14:50:50.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.440+0000 7fa69cbed640 1 -- 172.21.15.155:0/1179127270 >> 172.21.15.155:0/1179127270 conn(0x7fa6980b45c0 msgr2=0x7fa698142e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.441+0000 7f94bca23640 10 client.4320.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-09T14:50:50.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.441+0000 7f94bca23640 15 client.4320.objecter _session_command_op_assign -1 2 2024-08-09T14:50:50.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.441+0000 7f94bca23640 20 client.4320.objecter _get_session s=0x7f9498073580 osd=7 3 2024-08-09T14:50:50.473 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.441+0000 7f94bca23640 20 client.4320.objecter put_session s=0x7f9498073580 osd=7 3 2024-08-09T14:50:50.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.441+0000 7f94bca23640 20 client.4320.objecter _get_session s=0x7f9498073580 osd=7 3 2024-08-09T14:50:50.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.441+0000 7f94bca23640 15 client.4320.objecter _session_command_op_remove -1 2 2024-08-09T14:50:50.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.441+0000 7f94bca23640 20 client.4320.objecter get_session s=0x7f9498073580 osd=7 3 2024-08-09T14:50:50.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.441+0000 7f94bca23640 15 client.4320.objecter _session_command_op_assign 7 2 2024-08-09T14:50:50.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.441+0000 7f94bca23640 20 client.4320.objecter put_session s=0x7f9498073580 osd=7 4 2024-08-09T14:50:50.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.441+0000 7f94bca23640 10 client.4320.objecter _send_command 2 2024-08-09T14:50:50.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.441+0000 7f94bca23640 1 -- 172.21.15.155:0/976787639 --> [v2:172.21.15.194:6820/2670313675,v1:172.21.15.194:6823/2670313675] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f94b8072be0 con 0x7f9498073c50 2024-08-09T14:50:50.474 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.442+0000 7fa69cbed640 1 -- 172.21.15.155:0/1179127270 shutdown_connections 2024-08-09T14:50:50.475 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.442+0000 7fa69cbed640 1 -- 172.21.15.155:0/1179127270 wait complete. 2024-08-09T14:50:50.475 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.442+0000 7fa69cbed640 1 librados: shutdown 2024-08-09T14:50:50.475 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.443+0000 7fac527fc640 10 client.4323.objecter ms_handle_connect 0x7fac40073c50 2024-08-09T14:50:50.475 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.443+0000 7fac613b2640 1 --2- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] conn(0x7fac40073c50 0x7fac40076090 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.476 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.444+0000 7f20157fa640 1 -- 172.21.15.155:0/4032070454 <== osd.6 v2:172.21.15.194:6808/3077539270 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f200c006820 con 0x7f1ffc073c50 2024-08-09T14:50:50.476 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.444+0000 7f20157fa640 10 client.4317.objecter ms_dispatch 0x7f2018000b90 command_reply(tid 1: 0 ) 2024-08-09T14:50:50.476 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.444+0000 7f20157fa640 10 client.4317.objecter _finish_command 1 = system:0 2024-08-09T14:50:50.476 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.444+0000 7f20157fa640 20 client.4317.objecter put_session s=0x7f1ffc073580 osd=6 4 2024-08-09T14:50:50.476 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.444+0000 7f20157fa640 15 client.4317.objecter _session_command_op_remove 6 1 2024-08-09T14:50:50.476 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.446+0000 7f0bf78e1640 10 client.4277.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-09T14:50:50.476 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.446+0000 7f0bf78e1640 15 client.4277.objecter _session_command_op_assign -1 2 2024-08-09T14:50:50.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.446+0000 7f0bf78e1640 20 client.4277.objecter _get_session s=0x7f0bd4073530 osd=1 3 2024-08-09T14:50:50.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.446+0000 7f0bf78e1640 20 client.4277.objecter put_session s=0x7f0bd4073530 osd=1 3 2024-08-09T14:50:50.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.446+0000 7f0bf78e1640 20 client.4277.objecter _get_session s=0x7f0bd4073530 osd=1 3 2024-08-09T14:50:50.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.446+0000 7f0bf78e1640 15 client.4277.objecter _session_command_op_remove -1 2 2024-08-09T14:50:50.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.446+0000 7f0bf78e1640 20 client.4277.objecter get_session s=0x7f0bd4073530 osd=1 3 2024-08-09T14:50:50.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.446+0000 7f0bf78e1640 15 client.4277.objecter _session_command_op_assign 1 2 2024-08-09T14:50:50.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.446+0000 7f0bf78e1640 20 client.4277.objecter put_session s=0x7f0bd4073530 osd=1 4 2024-08-09T14:50:50.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.446+0000 7f0bf78e1640 10 client.4277.objecter _send_command 2 2024-08-09T14:50:50.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.446+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2103563403 --> [v2:172.21.15.155:6818/3129678288,v1:172.21.15.155:6819/3129678288] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0bf0146bf0 con 0x7f0bd4073c00 2024-08-09T14:50:50.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.448+0000 7f94aeffd640 1 -- 172.21.15.155:0/976787639 <== osd.7 v2:172.21.15.194:6820/2670313675 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f94b8072be0 con 0x7f9498073c50 2024-08-09T14:50:50.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.448+0000 7f94aeffd640 10 client.4320.objecter ms_dispatch 0x7f94b8000b90 command_reply(tid 2: 0 ) 2024-08-09T14:50:50.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.448+0000 7f94aeffd640 10 client.4320.objecter _finish_command 2 = system:0 2024-08-09T14:50:50.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.448+0000 7f94aeffd640 20 client.4320.objecter put_session s=0x7f9498073580 osd=7 4 2024-08-09T14:50:50.478 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.448+0000 7f94aeffd640 15 client.4320.objecter _session_command_op_remove 7 2 2024-08-09T14:50:50.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.448+0000 7f94bca23640 10 librados: watch_flush enter 2024-08-09T14:50:50.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.448+0000 7f94bca23640 10 librados: watch_flush exit 2024-08-09T14:50:50.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.448+0000 7f94bca23640 10 client.4320.objecter close_session for osd.7 2024-08-09T14:50:50.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.448+0000 7f94bca23640 1 -- 172.21.15.155:0/976787639 >> [v2:172.21.15.194:6820/2670313675,v1:172.21.15.194:6823/2670313675] conn(0x7f9498073c50 msgr2=0x7f9498076090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.448+0000 7f94bca23640 1 --2- 172.21.15.155:0/976787639 >> [v2:172.21.15.194:6820/2670313675,v1:172.21.15.194:6823/2670313675] conn(0x7f9498073c50 0x7f9498076090 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94bca23640 20 client.4320.objecter put_session s=0x7f9498073580 osd=7 1 2024-08-09T14:50:50.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94bca23640 20 client.4320.objecter shutdown clearing up homeless session... 2024-08-09T14:50:50.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94bca23640 10 client.4320.objecter successfully canceled tick 2024-08-09T14:50:50.479 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94bca23640 1 -- 172.21.15.155:0/976787639 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f949806d810 msgr2=0x7f949806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.480 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94bca23640 1 --2- 172.21.15.155:0/976787639 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f949806d810 0x7f949806fcd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f94b81450f0 tx=0x7f94a0006c90 comp rx=0 tx=0).stop 2024-08-09T14:50:50.480 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94bca23640 10 monclient: shutdown 2024-08-09T14:50:50.480 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94bca23640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.480 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94bca23640 1 -- 172.21.15.155:0/976787639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94b806b910 msgr2=0x7f94b81467f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.480 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94bca23640 1 --2- 172.21.15.155:0/976787639 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f94b806b910 0x7f94b81467f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f94a800b410 tx=0x7f94a800b8d0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.480 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94b6d9d640 1 -- 172.21.15.155:0/976787639 reap_dead start 2024-08-09T14:50:50.480 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94bca23640 1 -- 172.21.15.155:0/976787639 shutdown_connections 2024-08-09T14:50:50.480 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94bca23640 1 -- 172.21.15.155:0/976787639 >> 172.21.15.155:0/976787639 conn(0x7f94b80b2700 msgr2=0x7f94b806fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.480 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94bca23640 1 -- 172.21.15.155:0/976787639 shutdown_connections 2024-08-09T14:50:50.480 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94bca23640 1 -- 172.21.15.155:0/976787639 wait complete. 2024-08-09T14:50:50.480 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.449+0000 7f94bca23640 1 librados: shutdown 2024-08-09T14:50:50.481 INFO:teuthology.orchestra.run.smithi155.stdout:60129542152 2024-08-09T14:50:50.481 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-09T14:50:50.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.455+0000 7f0be5ffb640 1 -- 172.21.15.155:0/2103563403 <== osd.1 v2:172.21.15.155:6818/3129678288 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0bf0146bf0 con 0x7f0bd4073c00 2024-08-09T14:50:50.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.455+0000 7f0be5ffb640 10 client.4277.objecter ms_dispatch 0x7f0bf0000b90 command_reply(tid 2: 0 ) 2024-08-09T14:50:50.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.455+0000 7f0be5ffb640 10 client.4277.objecter _finish_command 2 = system:0 2024-08-09T14:50:50.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.455+0000 7f0be5ffb640 20 client.4277.objecter put_session s=0x7f0bd4073530 osd=1 4 2024-08-09T14:50:50.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.455+0000 7f0be5ffb640 15 client.4277.objecter _session_command_op_remove 1 2 2024-08-09T14:50:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 10 librados: watch_flush enter 2024-08-09T14:50:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 10 librados: watch_flush exit 2024-08-09T14:50:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 10 client.4277.objecter close_session for osd.1 2024-08-09T14:50:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:6818/3129678288,v1:172.21.15.155:6819/3129678288] conn(0x7f0bd4073c00 msgr2=0x7f0bd4076040 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 1 --2- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:6818/3129678288,v1:172.21.15.155:6819/3129678288] conn(0x7f0bd4073c00 0x7f0bd4076040 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 20 client.4277.objecter put_session s=0x7f0bd4073530 osd=1 1 2024-08-09T14:50:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 20 client.4277.objecter shutdown clearing up homeless session... 2024-08-09T14:50:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 10 client.4277.objecter successfully canceled tick 2024-08-09T14:50:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f0bd406d7c0 msgr2=0x7f0bd406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 1 --2- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f0bd406d7c0 0x7f0bd406fc80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0be0036000 tx=0x7f0be003d040 comp rx=0 tx=0).stop 2024-08-09T14:50:50.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 10 monclient: shutdown 2024-08-09T14:50:50.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bf0071010 msgr2=0x7f0bf0142c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 1 --2- 172.21.15.155:0/2103563403 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0bf0071010 0x7f0bf0142c80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00b4a0 tx=0x7f0bec00b960 comp rx=0 tx=0).stop 2024-08-09T14:50:50.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf5e7e640 1 -- 172.21.15.155:0/2103563403 reap_dead start 2024-08-09T14:50:50.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2103563403 shutdown_connections 2024-08-09T14:50:50.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.456+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2103563403 >> 172.21.15.155:0/2103563403 conn(0x7f0bf00b5af0 msgr2=0x7f0bf006b910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.457+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2103563403 shutdown_connections 2024-08-09T14:50:50.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.457+0000 7f0bf78e1640 1 -- 172.21.15.155:0/2103563403 wait complete. 2024-08-09T14:50:50.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.457+0000 7f0bf78e1640 1 librados: shutdown 2024-08-09T14:50:50.487 INFO:teuthology.orchestra.run.smithi155.stdout:85899345930 2024-08-09T14:50:50.487 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-09T14:50:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.464+0000 7f201edef640 10 client.4317.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-09T14:50:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.464+0000 7f201edef640 15 client.4317.objecter _session_command_op_assign -1 2 2024-08-09T14:50:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.464+0000 7f201edef640 20 client.4317.objecter _get_session s=0x7f1ffc073580 osd=6 3 2024-08-09T14:50:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.464+0000 7f201edef640 20 client.4317.objecter put_session s=0x7f1ffc073580 osd=6 3 2024-08-09T14:50:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.464+0000 7f201edef640 20 client.4317.objecter _get_session s=0x7f1ffc073580 osd=6 3 2024-08-09T14:50:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.464+0000 7f201edef640 15 client.4317.objecter _session_command_op_remove -1 2 2024-08-09T14:50:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.464+0000 7f201edef640 20 client.4317.objecter get_session s=0x7f1ffc073580 osd=6 3 2024-08-09T14:50:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.464+0000 7f201edef640 15 client.4317.objecter _session_command_op_assign 6 2 2024-08-09T14:50:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.464+0000 7f201edef640 20 client.4317.objecter put_session s=0x7f1ffc073580 osd=6 4 2024-08-09T14:50:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.464+0000 7f201edef640 10 client.4317.objecter _send_command 2 2024-08-09T14:50:50.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.464+0000 7f201edef640 1 -- 172.21.15.155:0/4032070454 --> [v2:172.21.15.194:6808/3077539270,v1:172.21.15.194:6809/3077539270] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f20180b8880 con 0x7f1ffc073c50 2024-08-09T14:50:50.489 INFO:teuthology.orchestra.run.smithi155.stdout:81604378635 2024-08-09T14:50:50.489 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-08-09T14:50:50.490 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.471+0000 7f20157fa640 1 -- 172.21.15.155:0/4032070454 <== osd.6 v2:172.21.15.194:6808/3077539270 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f20180b8880 con 0x7f1ffc073c50 2024-08-09T14:50:50.490 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.471+0000 7f20157fa640 10 client.4317.objecter ms_dispatch 0x7f2018000b90 command_reply(tid 2: 0 ) 2024-08-09T14:50:50.490 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.471+0000 7f20157fa640 10 client.4317.objecter _finish_command 2 = system:0 2024-08-09T14:50:50.490 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.471+0000 7f20157fa640 20 client.4317.objecter put_session s=0x7f1ffc073580 osd=6 4 2024-08-09T14:50:50.490 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.471+0000 7f20157fa640 15 client.4317.objecter _session_command_op_remove 6 2 2024-08-09T14:50:50.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.471+0000 7f201edef640 10 librados: watch_flush enter 2024-08-09T14:50:50.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 10 librados: watch_flush exit 2024-08-09T14:50:50.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 10 client.4317.objecter close_session for osd.6 2024-08-09T14:50:50.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 1 -- 172.21.15.155:0/4032070454 >> [v2:172.21.15.194:6808/3077539270,v1:172.21.15.194:6809/3077539270] conn(0x7f1ffc073c50 msgr2=0x7f1ffc076090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 1 --2- 172.21.15.155:0/4032070454 >> [v2:172.21.15.194:6808/3077539270,v1:172.21.15.194:6809/3077539270] conn(0x7f1ffc073c50 0x7f1ffc076090 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 20 client.4317.objecter put_session s=0x7f1ffc073580 osd=6 1 2024-08-09T14:50:50.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 20 client.4317.objecter shutdown clearing up homeless session... 2024-08-09T14:50:50.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 10 client.4317.objecter successfully canceled tick 2024-08-09T14:50:50.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 1 -- 172.21.15.155:0/4032070454 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f1ffc06d810 msgr2=0x7f1ffc06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 1 --2- 172.21.15.155:0/4032070454 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f1ffc06d810 0x7f1ffc06fcd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f20180aab00 tx=0x7f2004009110 comp rx=0 tx=0).stop 2024-08-09T14:50:50.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 10 monclient: shutdown 2024-08-09T14:50:50.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 1 -- 172.21.15.155:0/4032070454 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20180b0160 msgr2=0x7f20181ce4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 1 --2- 172.21.15.155:0/4032070454 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f20180b0160 0x7f20181ce4e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f200c00b4a0 tx=0x7f200c00b8a0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.492 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201d38c640 1 -- 172.21.15.155:0/4032070454 reap_dead start 2024-08-09T14:50:50.493 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 1 -- 172.21.15.155:0/4032070454 shutdown_connections 2024-08-09T14:50:50.493 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 1 -- 172.21.15.155:0/4032070454 >> 172.21.15.155:0/4032070454 conn(0x7f201806dbd0 msgr2=0x7f201813f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.493 INFO:teuthology.orchestra.run.smithi155.stdout:64424509447 2024-08-09T14:50:50.494 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-09T14:50:50.494 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 1 -- 172.21.15.155:0/4032070454 shutdown_connections 2024-08-09T14:50:50.494 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 1 -- 172.21.15.155:0/4032070454 wait complete. 2024-08-09T14:50:50.494 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.472+0000 7f201edef640 1 librados: shutdown 2024-08-09T14:50:50.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.474+0000 7f6c8ffff640 1 -- 172.21.15.155:0/578328842 <== osd.5 v2:172.21.15.194:6816/1013364343 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6c94077130 con 0x7f6c940745c0 2024-08-09T14:50:50.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.474+0000 7f6c8ffff640 10 client.4294.objecter ms_dispatch 0x7f6ca8000b90 command_reply(tid 1: 0 ) 2024-08-09T14:50:50.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.474+0000 7f6c8ffff640 10 client.4294.objecter _finish_command 1 = system:0 2024-08-09T14:50:50.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.474+0000 7f6c8ffff640 20 client.4294.objecter put_session s=0x7f6c94073ef0 osd=5 4 2024-08-09T14:50:50.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.474+0000 7f6c8ffff640 15 client.4294.objecter _session_command_op_remove 5 1 2024-08-09T14:50:50.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.479+0000 7f3d127fc640 1 -- 172.21.15.155:0/4090729798 <== osd.0 v2:172.21.15.155:6820/729861593 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3d18008f30 con 0x7f3d00073c80 2024-08-09T14:50:50.496 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.479+0000 7f3d127fc640 10 client.4288.objecter ms_dispatch 0x7f3d1c000b90 command_reply(tid 1: 0 ) 2024-08-09T14:50:50.496 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.479+0000 7f3d127fc640 10 client.4288.objecter _finish_command 1 = system:0 2024-08-09T14:50:50.496 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.479+0000 7f3d127fc640 20 client.4288.objecter put_session s=0x7f3d000735b0 osd=0 4 2024-08-09T14:50:50.496 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.479+0000 7f3d127fc640 15 client.4288.objecter _session_command_op_remove 0 1 2024-08-09T14:50:50.497 INFO:teuthology.orchestra.run.smithi155.stdout:60129542152 2024-08-09T14:50:50.497 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-08-09T14:50:50.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.490+0000 7f6cae0a0640 10 client.4294.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-09T14:50:50.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.490+0000 7f6cae0a0640 15 client.4294.objecter _session_command_op_assign -1 2 2024-08-09T14:50:50.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.490+0000 7f6cae0a0640 20 client.4294.objecter _get_session s=0x7f6c94073ef0 osd=5 3 2024-08-09T14:50:50.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.490+0000 7f6cae0a0640 20 client.4294.objecter put_session s=0x7f6c94073ef0 osd=5 3 2024-08-09T14:50:50.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.490+0000 7f6cae0a0640 20 client.4294.objecter _get_session s=0x7f6c94073ef0 osd=5 3 2024-08-09T14:50:50.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.490+0000 7f6cae0a0640 15 client.4294.objecter _session_command_op_remove -1 2 2024-08-09T14:50:50.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.490+0000 7f6cae0a0640 20 client.4294.objecter get_session s=0x7f6c94073ef0 osd=5 3 2024-08-09T14:50:50.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.490+0000 7f6cae0a0640 15 client.4294.objecter _session_command_op_assign 5 2 2024-08-09T14:50:50.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.490+0000 7f6cae0a0640 20 client.4294.objecter put_session s=0x7f6c94073ef0 osd=5 4 2024-08-09T14:50:50.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.490+0000 7f6cae0a0640 10 client.4294.objecter _send_command 2 2024-08-09T14:50:50.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.490+0000 7f6cae0a0640 1 -- 172.21.15.155:0/578328842 --> [v2:172.21.15.194:6816/1013364343,v1:172.21.15.194:6817/1013364343] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6ca8148d60 con 0x7f6c940745c0 2024-08-09T14:50:50.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.494+0000 7f3d22c22640 10 client.4288.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-09T14:50:50.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.494+0000 7f3d22c22640 15 client.4288.objecter _session_command_op_assign -1 2 2024-08-09T14:50:50.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.494+0000 7f3d22c22640 20 client.4288.objecter _get_session s=0x7f3d000735b0 osd=0 3 2024-08-09T14:50:50.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.494+0000 7f3d22c22640 20 client.4288.objecter put_session s=0x7f3d000735b0 osd=0 3 2024-08-09T14:50:50.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.494+0000 7f3d22c22640 20 client.4288.objecter _get_session s=0x7f3d000735b0 osd=0 3 2024-08-09T14:50:50.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.494+0000 7f3d22c22640 15 client.4288.objecter _session_command_op_remove -1 2 2024-08-09T14:50:50.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.494+0000 7f3d22c22640 20 client.4288.objecter get_session s=0x7f3d000735b0 osd=0 3 2024-08-09T14:50:50.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.494+0000 7f3d22c22640 15 client.4288.objecter _session_command_op_assign 0 2 2024-08-09T14:50:50.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.494+0000 7f3d22c22640 20 client.4288.objecter put_session s=0x7f3d000735b0 osd=0 4 2024-08-09T14:50:50.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.494+0000 7f3d22c22640 10 client.4288.objecter _send_command 2 2024-08-09T14:50:50.500 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.494+0000 7f3d22c22640 1 -- 172.21.15.155:0/4090729798 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3d14000b20 con 0x7f3d00073c80 2024-08-09T14:50:50.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6c8ffff640 1 -- 172.21.15.155:0/578328842 <== osd.5 v2:172.21.15.194:6816/1013364343 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6ca8148d60 con 0x7f6c940745c0 2024-08-09T14:50:50.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6c8ffff640 10 client.4294.objecter ms_dispatch 0x7f6ca8000b90 command_reply(tid 2: 0 ) 2024-08-09T14:50:50.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6c8ffff640 10 client.4294.objecter _finish_command 2 = system:0 2024-08-09T14:50:50.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6c8ffff640 20 client.4294.objecter put_session s=0x7f6c94073ef0 osd=5 4 2024-08-09T14:50:50.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6c8ffff640 15 client.4294.objecter _session_command_op_remove 5 2 2024-08-09T14:50:50.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 10 librados: watch_flush enter 2024-08-09T14:50:50.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 10 librados: watch_flush exit 2024-08-09T14:50:50.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 10 client.4294.objecter close_session for osd.5 2024-08-09T14:50:50.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 1 -- 172.21.15.155:0/578328842 >> [v2:172.21.15.194:6816/1013364343,v1:172.21.15.194:6817/1013364343] conn(0x7f6c940745c0 msgr2=0x7f6c94076a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 1 --2- 172.21.15.155:0/578328842 >> [v2:172.21.15.194:6816/1013364343,v1:172.21.15.194:6817/1013364343] conn(0x7f6c940745c0 0x7f6c94076a60 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 20 client.4294.objecter put_session s=0x7f6c94073ef0 osd=5 1 2024-08-09T14:50:50.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 20 client.4294.objecter shutdown clearing up homeless session... 2024-08-09T14:50:50.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 10 client.4294.objecter successfully canceled tick 2024-08-09T14:50:50.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 1 -- 172.21.15.155:0/578328842 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f6c9406d7c0 msgr2=0x7f6c9406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 1 --2- 172.21.15.155:0/578328842 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f6c9406d7c0 0x7f6c9406fc80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0000f80 tx=0x7f6ca0030000 comp rx=0 tx=0).stop 2024-08-09T14:50:50.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 10 monclient: shutdown 2024-08-09T14:50:50.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 1 -- 172.21.15.155:0/578328842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ca80ae9c0 msgr2=0x7f6ca81dc420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 1 --2- 172.21.15.155:0/578328842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ca80ae9c0 0x7f6ca81dc420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c00c3c0 tx=0x7f6c9c00c7c0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6ca7fff640 1 -- 172.21.15.155:0/578328842 reap_dead start 2024-08-09T14:50:50.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 1 -- 172.21.15.155:0/578328842 shutdown_connections 2024-08-09T14:50:50.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 1 -- 172.21.15.155:0/578328842 >> 172.21.15.155:0/578328842 conn(0x7f6ca80b45c0 msgr2=0x7f6ca80b17f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 1 -- 172.21.15.155:0/578328842 shutdown_connections 2024-08-09T14:50:50.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 1 -- 172.21.15.155:0/578328842 wait complete. 2024-08-09T14:50:50.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.496+0000 7f6cae0a0640 1 librados: shutdown 2024-08-09T14:50:50.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.502+0000 7fac527fc640 1 -- 172.21.15.155:0/2011517931 <== osd.3 v2:172.21.15.155:6810/3967029756 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fac54004810 con 0x7fac40073c50 2024-08-09T14:50:50.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.502+0000 7fac527fc640 10 client.4323.objecter ms_dispatch 0x7fac5c000b90 command_reply(tid 1: 0 ) 2024-08-09T14:50:50.503 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.502+0000 7fac527fc640 10 client.4323.objecter _finish_command 1 = system:0 2024-08-09T14:50:50.504 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.502+0000 7fac527fc640 20 client.4323.objecter put_session s=0x7fac40073580 osd=3 4 2024-08-09T14:50:50.504 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.502+0000 7fac527fc640 15 client.4323.objecter _session_command_op_remove 3 1 2024-08-09T14:50:50.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.506+0000 7f3d127fc640 1 -- 172.21.15.155:0/4090729798 <== osd.0 v2:172.21.15.155:6820/729861593 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3d14000b20 con 0x7f3d00073c80 2024-08-09T14:50:50.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.506+0000 7f3d127fc640 10 client.4288.objecter ms_dispatch 0x7f3d1c000b90 command_reply(tid 2: 0 ) 2024-08-09T14:50:50.508 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.506+0000 7f3d127fc640 10 client.4288.objecter _finish_command 2 = system:0 2024-08-09T14:50:50.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.506+0000 7f3d127fc640 20 client.4288.objecter put_session s=0x7f3d000735b0 osd=0 4 2024-08-09T14:50:50.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.506+0000 7f3d127fc640 15 client.4288.objecter _session_command_op_remove 0 2 2024-08-09T14:50:50.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.507+0000 7f3d22c22640 10 librados: watch_flush enter 2024-08-09T14:50:50.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.507+0000 7f3d22c22640 10 librados: watch_flush exit 2024-08-09T14:50:50.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.507+0000 7f3d22c22640 10 client.4288.objecter close_session for osd.0 2024-08-09T14:50:50.509 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.507+0000 7f3d22c22640 1 -- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] conn(0x7f3d00073c80 msgr2=0x7f3d000760c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.507+0000 7f3d22c22640 1 --2- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] conn(0x7f3d00073c80 0x7f3d000760c0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.507+0000 7f3d22c22640 20 client.4288.objecter put_session s=0x7f3d000735b0 osd=0 1 2024-08-09T14:50:50.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.507+0000 7f3d22c22640 20 client.4288.objecter shutdown clearing up homeless session... 2024-08-09T14:50:50.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.507+0000 7f3d22c22640 10 client.4288.objecter successfully canceled tick 2024-08-09T14:50:50.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.507+0000 7f3d22c22640 1 -- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f3d0006d840 msgr2=0x7f3d0006fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.507+0000 7f3d22c22640 1 --2- 172.21.15.155:0/4090729798 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f3d0006d840 0x7f3d0006fd00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c050f50 tx=0x7f3d1c0675c0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.507+0000 7f3d22c22640 10 monclient: shutdown 2024-08-09T14:50:50.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.507+0000 7f3d22c22640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.507+0000 7f3d22c22640 1 -- 172.21.15.155:0/4090729798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d140c34e0 msgr2=0x7f3d140c4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.507+0000 7f3d22c22640 1 --2- 172.21.15.155:0/4090729798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d140c34e0 0x7f3d140c4e40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3d18009870 tx=0x7f3d180046c0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.508+0000 7f3d22421640 1 -- 172.21.15.155:0/4090729798 reap_dead start 2024-08-09T14:50:50.510 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.508+0000 7f3d22c22640 1 -- 172.21.15.155:0/4090729798 shutdown_connections 2024-08-09T14:50:50.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.508+0000 7f3d22c22640 1 -- 172.21.15.155:0/4090729798 >> 172.21.15.155:0/4090729798 conn(0x7f3d140059f0 msgr2=0x7f3d14033370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.511 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.510+0000 7f3d22c22640 1 -- 172.21.15.155:0/4090729798 shutdown_connections 2024-08-09T14:50:50.512 INFO:teuthology.orchestra.run.smithi155.stdout:60129542150 2024-08-09T14:50:50.512 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-09T14:50:50.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.511+0000 7f3d22c22640 1 -- 172.21.15.155:0/4090729798 wait complete. 2024-08-09T14:50:50.513 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.511+0000 7f3d22c22640 1 librados: shutdown 2024-08-09T14:50:50.528 INFO:teuthology.orchestra.run.smithi155.stdout:90194313227 2024-08-09T14:50:50.529 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-09T14:50:50.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.530+0000 7fac63e17640 10 client.4323.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-08-09T14:50:50.531 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.530+0000 7fac63e17640 15 client.4323.objecter _session_command_op_assign -1 2 2024-08-09T14:50:50.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.530+0000 7fac63e17640 20 client.4323.objecter _get_session s=0x7fac40073580 osd=3 3 2024-08-09T14:50:50.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.530+0000 7fac63e17640 20 client.4323.objecter put_session s=0x7fac40073580 osd=3 3 2024-08-09T14:50:50.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.530+0000 7fac63e17640 20 client.4323.objecter _get_session s=0x7fac40073580 osd=3 3 2024-08-09T14:50:50.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.530+0000 7fac63e17640 15 client.4323.objecter _session_command_op_remove -1 2 2024-08-09T14:50:50.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.530+0000 7fac63e17640 20 client.4323.objecter get_session s=0x7fac40073580 osd=3 3 2024-08-09T14:50:50.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.530+0000 7fac63e17640 15 client.4323.objecter _session_command_op_assign 3 2 2024-08-09T14:50:50.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.530+0000 7fac63e17640 20 client.4323.objecter put_session s=0x7fac40073580 osd=3 4 2024-08-09T14:50:50.532 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.530+0000 7fac63e17640 10 client.4323.objecter _send_command 2 2024-08-09T14:50:50.533 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.530+0000 7fac63e17640 1 -- 172.21.15.155:0/2011517931 --> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fac5c143b10 con 0x7fac40073c50 2024-08-09T14:50:50.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.541+0000 7fac527fc640 1 -- 172.21.15.155:0/2011517931 <== osd.3 v2:172.21.15.155:6810/3967029756 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fac5c143b10 con 0x7fac40073c50 2024-08-09T14:50:50.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.541+0000 7fac527fc640 10 client.4323.objecter ms_dispatch 0x7fac5c000b90 command_reply(tid 2: 0 ) 2024-08-09T14:50:50.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.541+0000 7fac527fc640 10 client.4323.objecter _finish_command 2 = system:0 2024-08-09T14:50:50.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.541+0000 7fac527fc640 20 client.4323.objecter put_session s=0x7fac40073580 osd=3 4 2024-08-09T14:50:50.543 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.541+0000 7fac527fc640 15 client.4323.objecter _session_command_op_remove 3 2 2024-08-09T14:50:50.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.541+0000 7fac63e17640 10 librados: watch_flush enter 2024-08-09T14:50:50.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.541+0000 7fac63e17640 10 librados: watch_flush exit 2024-08-09T14:50:50.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.541+0000 7fac63e17640 10 client.4323.objecter close_session for osd.3 2024-08-09T14:50:50.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.541+0000 7fac63e17640 1 -- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] conn(0x7fac40073c50 msgr2=0x7fac40076090 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.541+0000 7fac63e17640 1 --2- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:6810/3967029756,v1:172.21.15.155:6811/3967029756] conn(0x7fac40073c50 0x7fac40076090 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.542+0000 7fac63e17640 20 client.4323.objecter put_session s=0x7fac40073580 osd=3 1 2024-08-09T14:50:50.544 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.542+0000 7fac63e17640 20 client.4323.objecter shutdown clearing up homeless session... 2024-08-09T14:50:50.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.542+0000 7fac63e17640 10 client.4323.objecter successfully canceled tick 2024-08-09T14:50:50.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.542+0000 7fac63e17640 1 -- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fac4006d810 msgr2=0x7fac4006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.542+0000 7fac63e17640 1 --2- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fac4006d810 0x7fac4006fcd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fac580090b0 tx=0x7fac5803d040 comp rx=0 tx=0).stop 2024-08-09T14:50:50.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.542+0000 7fac63e17640 10 monclient: shutdown 2024-08-09T14:50:50.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.542+0000 7fac63e17640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.542+0000 7fac63e17640 1 -- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac5c0b26e0 msgr2=0x7fac5c1495d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.542+0000 7fac63e17640 1 --2- 172.21.15.155:0/2011517931 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fac5c0b26e0 0x7fac5c1495d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fac5400efc0 tx=0x7fac5400c490 comp rx=0 tx=0).stop 2024-08-09T14:50:50.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.542+0000 7fac623b4640 1 -- 172.21.15.155:0/2011517931 reap_dead start 2024-08-09T14:50:50.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.542+0000 7fac63e17640 1 -- 172.21.15.155:0/2011517931 shutdown_connections 2024-08-09T14:50:50.545 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.542+0000 7fac63e17640 1 -- 172.21.15.155:0/2011517931 >> 172.21.15.155:0/2011517931 conn(0x7fac5c06dbd0 msgr2=0x7fac5c0b53f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.544+0000 7fac63e17640 1 -- 172.21.15.155:0/2011517931 shutdown_connections 2024-08-09T14:50:50.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.544+0000 7fac63e17640 1 -- 172.21.15.155:0/2011517931 wait complete. 2024-08-09T14:50:50.546 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.545+0000 7fac63e17640 1 librados: shutdown 2024-08-09T14:50:50.565 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2024-08-09T14:50:50.565 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-09T14:50:50.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.664+0000 7f9949f2c640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.664+0000 7f9949f2c640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.664+0000 7f9949f2c640 10 monclient: monmap: 2024-08-09T14:50:50.665 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.666 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.666 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.665203+0000 2024-08-09T14:50:50.666 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.665203+0000 2024-08-09T14:50:50.666 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.666 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.666 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.666 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.666 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.666 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.666 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.666 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.665+0000 7f9949f2c640 1 Processor -- start 2024-08-09T14:50:50.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.665+0000 7f9949f2c640 1 -- start start 2024-08-09T14:50:50.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.665+0000 7f9949f2c640 10 monclient: init 2024-08-09T14:50:50.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.665+0000 7f65cfc33640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.665+0000 7f65cfc33640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.668 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.665+0000 7f65cfc33640 10 monclient: monmap: 2024-08-09T14:50:50.668 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.668 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.668 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.666646+0000 2024-08-09T14:50:50.668 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.666646+0000 2024-08-09T14:50:50.669 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.669 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.669 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.669 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.669 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.669 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.669 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.669 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.671 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.666+0000 7f9949f2c640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.671 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.666+0000 7f9949f2c640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-08-09T14:50:50.671 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.666+0000 7f9949f2c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f994406d320 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:50.671 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f9949f2c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9944140510 0x7f99441460b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.671 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f9949f2c640 10 monclient(hunting): picked mon.noname-b con 0x7f9944140510 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.671 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f9949f2c640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f99441465f0 0x7f994414d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.671 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f9943fff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9944140510 0x7f99441460b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.671 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.668+0000 7f9949f2c640 10 monclient(hunting): picked mon.noname-a con 0x7f99441465f0 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:50.672 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.668+0000 7f9949f2c640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99440af880 con 0x7f994406d320 2024-08-09T14:50:50.672 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.668+0000 7f9949f2c640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.672 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.668+0000 7f9949f2c640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f994406d700 con 0x7f99441465f0 2024-08-09T14:50:50.672 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.668+0000 7f9949f2c640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.672 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.668+0000 7f9943fff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9944140510 0x7f99441460b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54688/0 (socket says 172.21.15.155:54688) 2024-08-09T14:50:50.672 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.666+0000 7f65cfc33640 1 Processor -- start 2024-08-09T14:50:50.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.666+0000 7f65cfc33640 1 -- start start 2024-08-09T14:50:50.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.666+0000 7f65cfc33640 10 monclient: init 2024-08-09T14:50:50.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f65cfc33640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f65cfc33640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-08-09T14:50:50.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f65cfc33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65c80b0160 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:50.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f65cfc33640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f65c80b4ef0 0x7f65c80b5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f65cfc33640 10 monclient(hunting): picked mon.noname-c con 0x7f65c80b4ef0 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:50.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f65cfc33640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65c81444e0 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:50.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f65cfc33640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65c80b30c0 con 0x7f65c80b0160 2024-08-09T14:50:50.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f65cfc33640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65c80b1890 con 0x7f65c81444e0 2024-08-09T14:50:50.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f65cfc33640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f65cfc33640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f65c80b0540 con 0x7f65c80b4ef0 2024-08-09T14:50:50.673 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f65cfc33640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.674 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.667+0000 7f65cfc33640 10 monclient(hunting): authenticate will time out at 1257.704938s 2024-08-09T14:50:50.675 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.668+0000 7f9943fff640 1 -- 172.21.15.155:0/1755589304 learned_addr learned my addr 172.21.15.155:0/1755589304 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.675 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.668+0000 7f9949f2c640 1 -- 172.21.15.155:0/1755589304 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f99440b13d0 con 0x7f9944140510 2024-08-09T14:50:50.675 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.668+0000 7f9949f2c640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.675 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.668+0000 7f9949f2c640 10 monclient(hunting): authenticate will time out at 1257.705642s 2024-08-09T14:50:50.675 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.668+0000 7f9943fff640 10 monclient(hunting): get_auth_request con 0x7f9944140510 auth_method 0 2024-08-09T14:50:50.675 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.668+0000 7f9943fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.675 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.668+0000 7f9943fff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.675 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.668+0000 7f9943fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.669+0000 7f65cd9cf640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f65c80b0160 0x7f65c80b1cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:34542/0 (socket says 172.21.15.155:34542) 2024-08-09T14:50:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.669+0000 7f65cd9cf640 1 -- 172.21.15.155:0/451257900 learned_addr learned my addr 172.21.15.155:0/451257900 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.669+0000 7f1b3937e640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.669+0000 7f1b3937e640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.669+0000 7f1b3937e640 10 monclient: monmap: 2024-08-09T14:50:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.676 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.670315+0000 2024-08-09T14:50:50.677 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.670315+0000 2024-08-09T14:50:50.677 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.677 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.677 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.677 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.677 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.677 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.677 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.677 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.678 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.669+0000 7f65cd1ce640 1 --2- 172.21.15.155:0/451257900 >> v2:172.21.15.194:3300/0 conn(0x7f65c80b4ef0 0x7f65c80b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.678 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.669+0000 7f65cd1ce640 10 monclient(hunting): get_auth_request con 0x7f65c80b4ef0 auth_method 0 2024-08-09T14:50:50.678 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.669+0000 7f65cd1ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.678 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.669+0000 7f65cd1ce640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.678 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.669+0000 7f65cd1ce640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.678 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.669+0000 7f9943fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.678 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.669+0000 7f9943fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.678 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.669+0000 7f9943fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.670+0000 7fb3b587b640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.670+0000 7fb3b587b640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.679 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.670+0000 7fb3b587b640 10 monclient: monmap: 2024-08-09T14:50:50.679 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.679 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.679 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.671164+0000 2024-08-09T14:50:50.679 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.671164+0000 2024-08-09T14:50:50.679 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.679 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.679 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.680 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.680 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.680 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.680 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.680 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.680 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f65cd1ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.680 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f65cd1ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.680 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f65cd1ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.681 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.670+0000 7f994972b640 1 --2- 172.21.15.155:0/1755589304 >> v2:172.21.15.155:3300/0 conn(0x7f99441465f0 0x7f994414d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.681 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f994972b640 10 monclient(hunting): get_auth_request con 0x7f99441465f0 auth_method 0 2024-08-09T14:50:50.681 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f994972b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.681 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f994972b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.681 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f994972b640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.670+0000 7f1b3937e640 1 Processor -- start 2024-08-09T14:50:50.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.670+0000 7f1b3937e640 1 -- start start 2024-08-09T14:50:50.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.670+0000 7f1b3937e640 10 monclient: init 2024-08-09T14:50:50.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f1b3937e640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f1b3937e640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-08-09T14:50:50.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.672+0000 7fb3b587b640 1 Processor -- start 2024-08-09T14:50:50.683 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f1b3937e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b3406d8d0 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:50.683 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f1b3937e640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f1b340b2d80 0x7f1b340aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.683 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f1b3937e640 10 monclient(hunting): picked mon.noname-c con 0x7f1b340b2d80 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:50.683 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f1b3937e640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f1b340b4df0 0x7f1b3413fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.683 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f1b3937e640 10 monclient(hunting): picked mon.noname-a con 0x7f1b340b4df0 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:50.683 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f1b3937e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b34143360 con 0x7f1b3406d8d0 2024-08-09T14:50:50.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f1b3937e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f1b3937e640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f1b3406dcb0 con 0x7f1b340b4df0 2024-08-09T14:50:50.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f1b3937e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f1b3937e640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f1b340b4680 con 0x7f1b340b2d80 2024-08-09T14:50:50.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f1b3937e640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.671+0000 7f1b3937e640 10 monclient(hunting): authenticate will time out at 1257.708985s 2024-08-09T14:50:50.685 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.672+0000 7fb3b587b640 1 -- start start 2024-08-09T14:50:50.685 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.672+0000 7fb3b587b640 10 monclient: init 2024-08-09T14:50:50.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.672+0000 7f9943fff640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2024-08-09T14:50:50.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.672+0000 7f9943fff640 1 -- 172.21.15.155:0/1755589304 >> v2:172.21.15.155:3300/0 conn(0x7f99441465f0 msgr2=0x7f994414d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.672+0000 7f9943fff640 1 --2- 172.21.15.155:0/1755589304 >> v2:172.21.15.155:3300/0 conn(0x7f99441465f0 0x7f994414d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.686 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.672+0000 7f9943fff640 1 -- 172.21.15.155:0/1755589304 >> v1:172.21.15.155:6789/0 conn(0x7f994406d320 legacy=0x7f994413fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7fb3b587b640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7fb3b587b640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-08-09T14:50:50.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7fb3b587b640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb3b0144240 0x7fb3b0071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7fb3b587b640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b0144240 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:50.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7fb3b587b640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b00b1150 0x7fb3b0144830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7fb3b587b640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b00b1150 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7fb3b587b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3b013dc30 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:50.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7fb3b587b640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3b0072c90 con 0x7fb3b013dc30 2024-08-09T14:50:50.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7fb3b587b640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.687 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7fb3b587b640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7fb3b0144620 con 0x7fb3b0144240 2024-08-09T14:50:50.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7fb3b587b640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7fb3b587b640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fb3b0071000 con 0x7fb3b00b1150 2024-08-09T14:50:50.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7fb3b587b640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7fb3b587b640 10 monclient(hunting): authenticate will time out at 1257.710135s 2024-08-09T14:50:50.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.672+0000 7f9943fff640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.688 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.672+0000 7f9943fff640 1 monclient: found mon.noname-b 2024-08-09T14:50:50.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.672+0000 7f9943fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.672+0000 7f9943fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.672+0000 7f9943fff640 1 -- 172.21.15.155:0/1755589304 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99440afc00 con 0x7f9944140510 2024-08-09T14:50:50.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7f65cd1ce640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2024-08-09T14:50:50.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7f65cd1ce640 1 -- 172.21.15.155:0/451257900 >> v1:172.21.15.194:6789/0 conn(0x7f65c81444e0 legacy=0x7f65c81519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7f65cd1ce640 1 -- 172.21.15.155:0/451257900 >> v1:172.21.15.155:6789/0 conn(0x7f65c80b0160 legacy=0x7f65c80b1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7f65cd1ce640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.689 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7f65cd1ce640 1 monclient: found mon.noname-c 2024-08-09T14:50:50.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7f65cd1ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7f65cd1ce640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.673+0000 7f65cd1ce640 1 -- 172.21.15.155:0/451257900 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c80b4680 con 0x7f65c80b4ef0 2024-08-09T14:50:50.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.674+0000 7f4417783640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.674+0000 7f4417783640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.674+0000 7f4417783640 10 monclient: monmap: 2024-08-09T14:50:50.690 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.690 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.690 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.675150+0000 2024-08-09T14:50:50.690 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.675150+0000 2024-08-09T14:50:50.691 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.691 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.691 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.691 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.691 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.691 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.692 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.692 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.692 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.674+0000 7fb3ae7fc640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b00b1150 0x7fb3b0144830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.692 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7f4417783640 1 Processor -- start 2024-08-09T14:50:50.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7fb3aeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fb3b0144240 0x7fb3b0071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7fb3ae7fc640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b00b1150 0x7fb3b0144830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54702/0 (socket says 172.21.15.155:54702) 2024-08-09T14:50:50.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7fb3ae7fc640 1 -- 172.21.15.155:0/1744668973 learned_addr learned my addr 172.21.15.155:0/1744668973 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7fb3ae7fc640 10 monclient(hunting): get_auth_request con 0x7fb3b00b1150 auth_method 0 2024-08-09T14:50:50.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7fb3ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7fb3ae7fc640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.693 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7fb3ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7f1b32ffd640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f1b3406d8d0 0x7f1b34142c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.155:40178/0 (socket says 172.21.15.155:40178) 2024-08-09T14:50:50.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7f1b32ffd640 1 -- 172.21.15.155:0/397361614 learned_addr learned my addr 172.21.15.155:0/397361614 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:50.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7fb3aeffd640 10 monclient(hunting): get_auth_request con 0x7fb3b0144240 auth_method 0 2024-08-09T14:50:50.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7fb3aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7fb3aeffd640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7fb3aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7f4417783640 1 -- start start 2024-08-09T14:50:50.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.675+0000 7f4417783640 10 monclient: init 2024-08-09T14:50:50.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7f4417783640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7f4417783640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-08-09T14:50:50.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7f4417783640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f441006bb80 0x7f4410070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7f4417783640 10 monclient(hunting): picked mon.noname-b con 0x7f441006bb80 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7f4417783640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f4410072fa0 0x7f44100733c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7f4417783640 10 monclient(hunting): picked mon.noname-c con 0x7f4410072fa0 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:50.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7f4417783640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44101470c0 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:50.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7f4417783640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4410070570 con 0x7f44101470c0 2024-08-09T14:50:50.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7f4417783640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7f4417783640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f441006bf60 con 0x7f4410072fa0 2024-08-09T14:50:50.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7f4417783640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7f4417783640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f44100b1f60 con 0x7f441006bb80 2024-08-09T14:50:50.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7f4417783640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.696 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7f4417783640 10 monclient(hunting): authenticate will time out at 1257.713364s 2024-08-09T14:50:50.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7fb3ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7fb3ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.676+0000 7fb3ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.677+0000 7fb3aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.677+0000 7fb3aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.677+0000 7fb3aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.697 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.677+0000 7f1b327fc640 1 --2- 172.21.15.155:0/397361614 >> v2:172.21.15.194:3300/0 conn(0x7f1b340b2d80 0x7f1b340aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.698 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.677+0000 7f1b327fc640 10 monclient(hunting): get_auth_request con 0x7f1b340b2d80 auth_method 0 2024-08-09T14:50:50.698 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.677+0000 7f1b327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.698 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.677+0000 7f1b327fc640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.698 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.677+0000 7f1b327fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.698 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.678+0000 7fb3ae7fc640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-08-09T14:50:50.698 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.678+0000 7fb3ae7fc640 1 -- 172.21.15.155:0/1744668973 >> v2:172.21.15.194:3300/0 conn(0x7fb3b0144240 msgr2=0x7fb3b0071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.698 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.678+0000 7fb3ae7fc640 1 --2- 172.21.15.155:0/1744668973 >> v2:172.21.15.194:3300/0 conn(0x7fb3b0144240 0x7fb3b0071a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.698 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.678+0000 7fb3ae7fc640 1 -- 172.21.15.155:0/1744668973 >> v1:172.21.15.194:6789/0 conn(0x7fb3b013dc30 legacy=0x7fb3b013e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.699 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.678+0000 7fb3ae7fc640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.699 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.678+0000 7fb3ae7fc640 1 monclient: found mon.noname-b 2024-08-09T14:50:50.699 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.678+0000 7fb3ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.699 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.678+0000 7fb3ae7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.699 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.678+0000 7fb3ae7fc640 1 -- 172.21.15.155:0/1744668973 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3b0072f70 con 0x7fb3b00b1150 2024-08-09T14:50:50.699 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.680+0000 7f9943fff640 1 --2- 172.21.15.155:0/1755589304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9944140510 0x7f99441460b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f994406df70 tx=0x7f993002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2167286971800ee9 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.700 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.681+0000 7f1b327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.700 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.681+0000 7f1b327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.700 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.681+0000 7f1b327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.700 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.681+0000 7f1b337fe640 1 --2- 172.21.15.155:0/397361614 >> v2:172.21.15.155:3300/0 conn(0x7f1b340b4df0 0x7f1b3413fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.700 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.682+0000 7f1b337fe640 10 monclient(hunting): get_auth_request con 0x7f1b340b4df0 auth_method 0 2024-08-09T14:50:50.700 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.682+0000 7f1b337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.700 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.682+0000 7f1b337fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.700 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.682+0000 7f1b337fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.701 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.682+0000 7fb3ae7fc640 1 --2- 172.21.15.155:0/1744668973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b00b1150 0x7fb3b0144830 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009e50 tx=0x7fb3a802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcc7b373c2910a5d server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.701 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.682+0000 7f1b337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.701 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.682+0000 7f1b337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.701 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.682+0000 7f1b337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.701 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.684+0000 7f1b327fc640 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2024-08-09T14:50:50.702 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.684+0000 7f1b327fc640 1 -- 172.21.15.155:0/397361614 >> v2:172.21.15.155:3300/0 conn(0x7f1b340b4df0 msgr2=0x7f1b3413fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.702 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.684+0000 7f1b327fc640 1 --2- 172.21.15.155:0/397361614 >> v2:172.21.15.155:3300/0 conn(0x7f1b340b4df0 0x7f1b3413fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.702 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.684+0000 7f1b327fc640 1 -- 172.21.15.155:0/397361614 >> v1:172.21.15.194:6789/0 conn(0x7f1b3406d8d0 legacy=0x7f1b34142c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.702 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.684+0000 7f1b327fc640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.702 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.684+0000 7f1b327fc640 1 monclient: found mon.noname-c 2024-08-09T14:50:50.702 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.684+0000 7f1b327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.702 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.684+0000 7f1b327fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.702 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.684+0000 7f1b327fc640 1 -- 172.21.15.155:0/397361614 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b34140530 con 0x7f1b340b2d80 2024-08-09T14:50:50.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.685+0000 7fb3adffb640 1 -- 172.21.15.155:0/1744668973 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a802c9e0 con 0x7fb3b00b1150 2024-08-09T14:50:50.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.685+0000 7fb3adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.703 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.685+0000 7f65cd1ce640 1 --2- 172.21.15.155:0/451257900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f65c80b4ef0 0x7f65c80b5760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f65b80097a0 tx=0x7f65b802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6daadbba21fe30a server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.685+0000 7fb3adffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:50.704 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.685+0000 7fb3adffb640 10 monclient: dump: 2024-08-09T14:50:50.704 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.704 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.704 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.704 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.705 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.705 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.705 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.705 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.705 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.705 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.685+0000 7fb3adffb640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.685+0000 7fb3adffb640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.685+0000 7fb3adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.686+0000 7fb3b587b640 5 monclient: authenticate success, global_id 4292 2024-08-09T14:50:50.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.686+0000 7fb3b587b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:50.705 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.688+0000 7fb3adffb640 1 -- 172.21.15.155:0/1744668973 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3a802cb80 con 0x7fb3b00b1150 2024-08-09T14:50:50.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.688+0000 7fb3adffb640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.688+0000 7f441551f640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f441006bb80 0x7f4410070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.706 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.688+0000 7f4414d1e640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f4410072fa0 0x7f44100733c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.688+0000 7fb3b587b640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:50.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.688+0000 7fb3b587b640 4 set_mon_vals no callback set 2024-08-09T14:50:50.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.689+0000 7fb3b587b640 10 monclient: shutdown 2024-08-09T14:50:50.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.689+0000 7fb3b587b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.689+0000 7fb3b587b640 1 -- 172.21.15.155:0/1744668973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b00b1150 msgr2=0x7fb3b0144830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.689+0000 7fb3b587b640 1 --2- 172.21.15.155:0/1744668973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b00b1150 0x7fb3b0144830 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009e50 tx=0x7fb3a802be40 comp rx=0 tx=0).stop 2024-08-09T14:50:50.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.689+0000 7fb3b587b640 1 -- 172.21.15.155:0/1744668973 shutdown_connections 2024-08-09T14:50:50.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.689+0000 7fb3b587b640 1 --2- 172.21.15.155:0/1744668973 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b00b1150 0x7fb3b0144830 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.689+0000 7fb3b587b640 1 --2- 172.21.15.155:0/1744668973 >> v2:172.21.15.194:3300/0 conn(0x7fb3b0144240 0x7fb3b0071a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.707 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.689+0000 7fb3b587b640 1 -- 172.21.15.155:0/1744668973 >> 172.21.15.155:0/1744668973 conn(0x7fb3b00680f0 msgr2=0x7fb3b00b6750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.689+0000 7fb3b587b640 1 -- 172.21.15.155:0/1744668973 shutdown_connections 2024-08-09T14:50:50.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.689+0000 7fb3b587b640 1 -- 172.21.15.155:0/1744668973 wait complete. 2024-08-09T14:50:50.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.689+0000 7f4415d20640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f44101470c0 0x7f4410147510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.155:40202/0 (socket says 172.21.15.155:40202) 2024-08-09T14:50:50.708 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.689+0000 7f4415d20640 1 -- 172.21.15.155:0/2791730943 learned_addr learned my addr 172.21.15.155:0/2791730943 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:50.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.690+0000 7fb3b587b640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.709 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.690+0000 7fb3b587b640 10 monclient: monmap: 2024-08-09T14:50:50.709 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.709 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:50.709 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:50.709 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:50.709 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.710 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.710 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:50.710 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:50.710 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:50.710 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.710 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.690+0000 7fb3b587b640 1 librados: starting msgr at 2024-08-09T14:50:50.710 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.690+0000 7fb3b587b640 1 librados: starting objecter 2024-08-09T14:50:50.710 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.691+0000 7f4414d1e640 10 monclient(hunting): get_auth_request con 0x7f4410072fa0 auth_method 0 2024-08-09T14:50:50.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.691+0000 7f4414d1e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.691+0000 7f4414d1e640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.711 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.691+0000 7f4414d1e640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.691+0000 7fb3b587b640 1 Processor -- start 2024-08-09T14:50:50.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.691+0000 7fb3b587b640 1 -- start start 2024-08-09T14:50:50.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.691+0000 7fb3b587b640 1 librados: setting wanted keys 2024-08-09T14:50:50.713 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.691+0000 7fb3b587b640 1 librados: calling monclient init 2024-08-09T14:50:50.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.691+0000 7fb3b587b640 10 monclient: init 2024-08-09T14:50:50.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 10 monclient: _add_conns ranks=[2,0,1] 2024-08-09T14:50:50.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b00b1150 0x7fb3b00b5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b00b1150 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b013dc30 0x7fb3b0145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b013dc30 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:50.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3b0144240 0x7fb3b01410f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b0144240 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:50.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b014b340 con 0x7fb3b013dc30 2024-08-09T14:50:50.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.714 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b0144d70 con 0x7fb3b0144240 2024-08-09T14:50:50.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fb3b014b4c0 con 0x7fb3b00b1150 2024-08-09T14:50:50.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7fb3b587b640 10 monclient(hunting): authenticate will time out at 1257.729275s 2024-08-09T14:50:50.715 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.692+0000 7f1b327fc640 1 --2- 172.21.15.155:0/397361614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b340b2d80 0x7f1b340aa9d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1b34143d10 tx=0x7f1b2802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4854982e3254d816 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.693+0000 7fb3ae7fc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b013dc30 0x7fb3b0145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.693+0000 7fb3ae7fc640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b013dc30 0x7fb3b0145710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43566/0 (socket says 172.21.15.155:43566) 2024-08-09T14:50:50.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.693+0000 7fb3ae7fc640 1 -- 172.21.15.155:0/4152612314 learned_addr learned my addr 172.21.15.155:0/4152612314 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.693+0000 7fb3ae7fc640 10 monclient(hunting): get_auth_request con 0x7fb3b013dc30 auth_method 0 2024-08-09T14:50:50.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.693+0000 7fb3ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.693+0000 7fb3ae7fc640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.693+0000 7fb3ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.694+0000 7fb3ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.694+0000 7fb3ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.694+0000 7fb3ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.716 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f441551f640 10 monclient(hunting): get_auth_request con 0x7f441006bb80 auth_method 0 2024-08-09T14:50:50.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f441551f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f441551f640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f441551f640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f99437fe640 1 -- 172.21.15.155:0/1755589304 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9930035020 con 0x7f9944140510 2024-08-09T14:50:50.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f99437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f99437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:50.717 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f99437fe640 10 monclient: dump: 2024-08-09T14:50:50.717 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.717 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.718 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.718 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.718 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.718 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.718 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.718 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.718 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.718 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f99437fe640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f99437fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f99437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f99437fe640 1 -- 172.21.15.155:0/1755589304 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9930031640 con 0x7f9944140510 2024-08-09T14:50:50.718 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f99437fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f99437fe640 1 -- 172.21.15.155:0/1755589304 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993002d040 con 0x7f9944140510 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f99437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f99437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f99437fe640 10 monclient: dump: 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.719 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f9949f2c640 5 monclient: authenticate success, global_id 4289 2024-08-09T14:50:50.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f9949f2c640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:50.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.695+0000 7f9949f2c640 4 set_mon_vals no callback set 2024-08-09T14:50:50.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.696+0000 7fb3af7fe640 1 --2- 172.21.15.155:0/4152612314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3b0144240 0x7fb3b01410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.696+0000 7f9949f2c640 10 monclient: shutdown 2024-08-09T14:50:50.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.696+0000 7f9949f2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.696+0000 7f9949f2c640 1 -- 172.21.15.155:0/1755589304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9944140510 msgr2=0x7f99441460b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.696+0000 7f9949f2c640 1 --2- 172.21.15.155:0/1755589304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9944140510 0x7f99441460b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f994406df70 tx=0x7f993002b8f0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.696+0000 7f9949f2c640 1 -- 172.21.15.155:0/1755589304 shutdown_connections 2024-08-09T14:50:50.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.696+0000 7f9949f2c640 1 --2- 172.21.15.155:0/1755589304 >> v2:172.21.15.155:3300/0 conn(0x7f99441465f0 0x7f994414d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.696+0000 7f9949f2c640 1 --2- 172.21.15.155:0/1755589304 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f9944140510 0x7f99441460b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.696+0000 7f9949f2c640 1 -- 172.21.15.155:0/1755589304 >> 172.21.15.155:0/1755589304 conn(0x7f9944068300 msgr2=0x7f99440b8730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.696+0000 7f9949f2c640 1 -- 172.21.15.155:0/1755589304 shutdown_connections 2024-08-09T14:50:50.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7f441551f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.696+0000 7f9949f2c640 1 -- 172.21.15.155:0/1755589304 wait complete. 2024-08-09T14:50:50.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7f441551f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.722 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7f441551f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7fb3ae7fc640 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2024-08-09T14:50:50.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7fb3ae7fc640 1 -- 172.21.15.155:0/4152612314 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b00b1150 msgr2=0x7fb3b00b5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7fb3ae7fc640 1 --2- 172.21.15.155:0/4152612314 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b00b1150 0x7fb3b00b5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7fb3ae7fc640 1 -- 172.21.15.155:0/4152612314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3b0144240 msgr2=0x7fb3b01410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7fb3ae7fc640 1 --2- 172.21.15.155:0/4152612314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3b0144240 0x7fb3b01410f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7fb3ae7fc640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7fb3ae7fc640 1 monclient: found mon.noname-a 2024-08-09T14:50:50.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7fb3ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7fb3ae7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7fb3ae7fc640 1 -- 172.21.15.155:0/4152612314 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3b0146b10 con 0x7fb3b013dc30 2024-08-09T14:50:50.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7f9949f2c640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7f9949f2c640 10 monclient: monmap: 2024-08-09T14:50:50.725 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.726 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:50.726 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:50.726 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:50.726 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.726 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.726 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:50.727 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:50.727 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:50.727 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7f9949f2c640 1 librados: starting msgr at 2024-08-09T14:50:50.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.697+0000 7f9949f2c640 1 librados: starting objecter 2024-08-09T14:50:50.727 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.698+0000 7f9949f2c640 1 Processor -- start 2024-08-09T14:50:50.728 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.698+0000 7f9949f2c640 1 -- start start 2024-08-09T14:50:50.728 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.698+0000 7f9949f2c640 1 librados: setting wanted keys 2024-08-09T14:50:50.728 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.698+0000 7f9949f2c640 1 librados: calling monclient init 2024-08-09T14:50:50.728 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.698+0000 7f9949f2c640 10 monclient: init 2024-08-09T14:50:50.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.698+0000 7f4414d1e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.698+0000 7f4414d1e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.698+0000 7f4414d1e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b31ffb640 1 -- 172.21.15.155:0/397361614 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28035020 con 0x7f1b340b2d80 2024-08-09T14:50:50.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b31ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:50.731 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b31ffb640 10 monclient: dump: 2024-08-09T14:50:50.732 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.732 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.732 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.732 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.732 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.732 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.733 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.733 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.733 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.733 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.733 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b31ffb640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.734 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b31ffb640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.734 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.734 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b31ffb640 1 -- 172.21.15.155:0/397361614 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b28031830 con 0x7f1b340b2d80 2024-08-09T14:50:50.734 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.735 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b31ffb640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.735 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b31ffb640 1 -- 172.21.15.155:0/397361614 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28031c00 con 0x7f1b340b2d80 2024-08-09T14:50:50.735 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.735 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b31ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:50.736 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b31ffb640 10 monclient: dump: 2024-08-09T14:50:50.736 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.736 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.736 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.736 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.736 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.737 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.737 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.737 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.737 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.737 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.737 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b3937e640 5 monclient: authenticate success, global_id 4303 2024-08-09T14:50:50.738 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b3937e640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:50.738 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f1b3937e640 4 set_mon_vals no callback set 2024-08-09T14:50:50.738 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 10 monclient: _add_conns ranks=[2,1,0] 2024-08-09T14:50:50.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994406d320 0x7f994413f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 10 monclient(hunting): picked mon.noname-c con 0x7f994406d320 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9944140510 0x7f994413aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 10 monclient(hunting): picked mon.noname-b con 0x7f9944140510 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:50.739 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99441465f0 0x7f994413b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 10 monclient(hunting): picked mon.noname-a con 0x7f99441465f0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:50.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f9944146c20 con 0x7f99441465f0 2024-08-09T14:50:50.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.740 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9944146aa0 con 0x7f9944140510 2024-08-09T14:50:50.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f99440b4ad0 con 0x7f994406d320 2024-08-09T14:50:50.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.741 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.699+0000 7f9949f2c640 10 monclient(hunting): authenticate will time out at 1257.736295s 2024-08-09T14:50:50.742 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.700+0000 7f1b3937e640 10 monclient: shutdown 2024-08-09T14:50:50.742 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.700+0000 7f6e93faf640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.700+0000 7f6e93faf640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.700+0000 7f6e93faf640 10 monclient: monmap: 2024-08-09T14:50:50.743 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.743 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.743 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.701145+0000 2024-08-09T14:50:50.744 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.701145+0000 2024-08-09T14:50:50.744 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.744 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.744 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.744 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.745 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.745 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.745 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.745 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.700+0000 7f1b3937e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.700+0000 7f1b3937e640 1 -- 172.21.15.155:0/397361614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b340b2d80 msgr2=0x7f1b340aa9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.747 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.700+0000 7f1b3937e640 1 --2- 172.21.15.155:0/397361614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b340b2d80 0x7f1b340aa9d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1b34143d10 tx=0x7f1b2802b8d0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.700+0000 7f1b3937e640 1 -- 172.21.15.155:0/397361614 shutdown_connections 2024-08-09T14:50:50.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.700+0000 7f1b3937e640 1 --2- 172.21.15.155:0/397361614 >> v2:172.21.15.155:3300/0 conn(0x7f1b340b4df0 0x7f1b3413fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.700+0000 7f1b3937e640 1 --2- 172.21.15.155:0/397361614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b340b2d80 0x7f1b340aa9d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.700+0000 7f1b3937e640 1 -- 172.21.15.155:0/397361614 >> 172.21.15.155:0/397361614 conn(0x7f1b340b8d70 msgr2=0x7f1b340b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.748 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.700+0000 7f1b3937e640 1 -- 172.21.15.155:0/397361614 shutdown_connections 2024-08-09T14:50:50.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.700+0000 7f1b3937e640 1 -- 172.21.15.155:0/397361614 wait complete. 2024-08-09T14:50:50.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.700+0000 7f6e93faf640 1 Processor -- start 2024-08-09T14:50:50.749 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f6e93faf640 1 -- start start 2024-08-09T14:50:50.750 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f6e93faf640 10 monclient: init 2024-08-09T14:50:50.750 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f994972b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99441465f0 0x7f994413b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f1b3937e640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.751 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f1b3937e640 10 monclient: monmap: 2024-08-09T14:50:50.751 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.751 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:50.751 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:50.751 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:50.751 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.752 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.752 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:50.752 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:50.752 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:50.752 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.753 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f441551f640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2024-08-09T14:50:50.753 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f441551f640 1 -- 172.21.15.155:0/2791730943 >> v2:172.21.15.194:3300/0 conn(0x7f4410072fa0 msgr2=0x7f44100733c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.753 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f441551f640 1 --2- 172.21.15.155:0/2791730943 >> v2:172.21.15.194:3300/0 conn(0x7f4410072fa0 0x7f44100733c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.753 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f441551f640 1 -- 172.21.15.155:0/2791730943 >> v1:172.21.15.194:6789/0 conn(0x7f44101470c0 legacy=0x7f4410147510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.753 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f441551f640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.753 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f441551f640 1 monclient: found mon.noname-b 2024-08-09T14:50:50.753 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f441551f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.754 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f441551f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.754 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f441551f640 1 -- 172.21.15.155:0/2791730943 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f441006f7e0 con 0x7f441006bb80 2024-08-09T14:50:50.754 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f1b3937e640 1 librados: starting msgr at 2024-08-09T14:50:50.754 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f1b3937e640 1 librados: starting objecter 2024-08-09T14:50:50.755 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f994972b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99441465f0 0x7f994413b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43580/0 (socket says 172.21.15.155:43580) 2024-08-09T14:50:50.755 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f994972b640 1 -- 172.21.15.155:0/2389289502 learned_addr learned my addr 172.21.15.155:0/2389289502 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.755 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f994972b640 10 monclient(hunting): get_auth_request con 0x7f99441465f0 auth_method 0 2024-08-09T14:50:50.755 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f994972b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.755 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f994972b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.755 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f994972b640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f6e93faf640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f6e93faf640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-08-09T14:50:50.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f6e93faf640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f6e8c0b0160 0x7f6e8c0b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f6e93faf640 10 monclient(hunting): picked mon.noname-c con 0x7f6e8c0b0160 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:50.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f6e93faf640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f6e8c0b22e0 0x7f6e8c1469d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f6e93faf640 10 monclient(hunting): picked mon.noname-a con 0x7f6e8c0b22e0 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:50.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.701+0000 7f6e93faf640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e8c146f10 0x7f6e8c0b3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f6e93faf640 10 monclient(hunting): picked mon.noname-b con 0x7f6e8c146f10 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f1b3937e640 1 Processor -- start 2024-08-09T14:50:50.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f6e93faf640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f6e93faf640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f6e8c1461c0 con 0x7f6e8c0b22e0 2024-08-09T14:50:50.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f6e93faf640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f6e93faf640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f6e8c0b26e0 con 0x7f6e8c0b0160 2024-08-09T14:50:50.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f6e93faf640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f6e93faf640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f6e8c0b2890 con 0x7f6e8c146f10 2024-08-09T14:50:50.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f6e93faf640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f6e93faf640 10 monclient(hunting): authenticate will time out at 1257.739040s 2024-08-09T14:50:50.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f1b3937e640 1 -- start start 2024-08-09T14:50:50.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f1b3937e640 1 librados: setting wanted keys 2024-08-09T14:50:50.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f1b3937e640 1 librados: calling monclient init 2024-08-09T14:50:50.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f1b3937e640 10 monclient: init 2024-08-09T14:50:50.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f994972b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f994972b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7f994972b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7fdac5c5b640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.702+0000 7fdac5c5b640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7fdac5c5b640 10 monclient: monmap: 2024-08-09T14:50:50.760 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.760 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.760 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.704042+0000 2024-08-09T14:50:50.760 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.704042+0000 2024-08-09T14:50:50.760 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.760 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.760 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.760 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.760 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.761 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.761 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.761 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 10 monclient: _add_conns ranks=[2,1,0] 2024-08-09T14:50:50.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b3406d8d0 0x7f1b340701f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 10 monclient(hunting): picked mon.noname-c con 0x7f1b3406d8d0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b340b2d80 0x7f1b34145200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 10 monclient(hunting): picked mon.noname-b con 0x7f1b340b2d80 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:50.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7fb3ae7fc640 1 --2- 172.21.15.155:0/4152612314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b013dc30 0x7fb3b0145710 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009f70 tx=0x7fb3a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7fb397fff640 10 client.?.objecter ms_handle_connect 0x7fb3b013dc30 2024-08-09T14:50:50.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7fb397fff640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:50.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b340b4df0 0x7f1b34145740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 10 monclient(hunting): picked mon.noname-a con 0x7f1b340b4df0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:50.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f1b341408a0 con 0x7f1b340b4df0 2024-08-09T14:50:50.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1b340b4960 con 0x7f1b340b2d80 2024-08-09T14:50:50.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f1b3414d6f0 con 0x7f1b3406d8d0 2024-08-09T14:50:50.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.703+0000 7f1b3937e640 10 monclient(hunting): authenticate will time out at 1257.740429s 2024-08-09T14:50:50.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.704+0000 7fdac5c5b640 1 Processor -- start 2024-08-09T14:50:50.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.704+0000 7fdac5c5b640 1 -- start start 2024-08-09T14:50:50.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.704+0000 7fdac5c5b640 10 monclient: init 2024-08-09T14:50:50.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7fdac5c5b640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7fdac5c5b640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-08-09T14:50:50.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7fdac5c5b640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fdab8039210 0x7fdab8035860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7fdac5c5b640 10 monclient(hunting): picked mon.noname-a con 0x7fdab8039210 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:50.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7f9948f2a640 1 --2- 172.21.15.155:0/2389289502 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994406d320 0x7f994413f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7fdac5c5b640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fdab80c46d0 0x7fdab80c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7fdac5c5b640 10 monclient(hunting): picked mon.noname-c con 0x7fdab80c46d0 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:50.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7fdac5c5b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdab80cac70 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:50.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7fdac5c5b640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdab80c5030 con 0x7fdab80cac70 2024-08-09T14:50:50.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7fdac5c5b640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7fdac5c5b640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7fdab80395f0 con 0x7fdab8039210 2024-08-09T14:50:50.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7fdac5c5b640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7fdac5c5b640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7fdab8030a50 con 0x7fdab80c46d0 2024-08-09T14:50:50.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7fdac5c5b640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7fdac5c5b640 10 monclient(hunting): authenticate will time out at 1257.742445s 2024-08-09T14:50:50.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7f9948f2a640 10 monclient(hunting): get_auth_request con 0x7f994406d320 auth_method 0 2024-08-09T14:50:50.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7f9948f2a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7f9948f2a640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.705+0000 7f9948f2a640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.706+0000 7f6e91d4b640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f6e8c0b0160 0x7f6e8c0b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.706+0000 7f65cc9cd640 1 -- 172.21.15.155:0/451257900 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b8034070 con 0x7f65c80b4ef0 2024-08-09T14:50:50.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.706+0000 7f65cc9cd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.706+0000 7f65cc9cd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:50.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.706+0000 7f65cc9cd640 10 monclient: dump: 2024-08-09T14:50:50.768 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.768 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.768 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.768 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.768 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.769 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.769 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.769 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.769 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.769 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.706+0000 7f65cc9cd640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.706+0000 7f65cc9cd640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.706+0000 7f65cc9cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.706+0000 7f65cc9cd640 1 -- 172.21.15.155:0/451257900 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65b8030610 con 0x7f65c80b4ef0 2024-08-09T14:50:50.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.706+0000 7f65cfc33640 5 monclient: authenticate success, global_id 4297 2024-08-09T14:50:50.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.706+0000 7f65cfc33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:50.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.706+0000 7f65cc9cd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.706+0000 7f65cfc33640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:50.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.706+0000 7f65cfc33640 4 set_mon_vals no callback set 2024-08-09T14:50:50.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f9948f2a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f9948f2a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f9948f2a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f994972b640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-08-09T14:50:50.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f994972b640 1 -- 172.21.15.155:0/2389289502 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994406d320 msgr2=0x7f994413f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f994972b640 1 --2- 172.21.15.155:0/2389289502 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994406d320 0x7f994413f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f994972b640 1 -- 172.21.15.155:0/2389289502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9944140510 msgr2=0x7f994413aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f994972b640 1 --2- 172.21.15.155:0/2389289502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9944140510 0x7f994413aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f994972b640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f994972b640 1 monclient: found mon.noname-a 2024-08-09T14:50:50.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f994972b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f994972b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f994972b640 1 -- 172.21.15.155:0/2389289502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f994413e7e0 con 0x7f99441465f0 2024-08-09T14:50:50.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f65cc9cd640 1 -- 172.21.15.155:0/451257900 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b80309e0 con 0x7f65c80b4ef0 2024-08-09T14:50:50.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f65cc9cd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f65cc9cd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:50.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f65cc9cd640 10 monclient: dump: 2024-08-09T14:50:50.772 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.772 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.772 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.772 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.772 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.772 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.772 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.772 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.772 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.772 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f6e91d4b640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f6e8c0b0160 0x7f6e8c0b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:47922/0 (socket says 172.21.15.155:47922) 2024-08-09T14:50:50.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f6e91d4b640 1 -- 172.21.15.155:0/364099162 learned_addr learned my addr 172.21.15.155:0/364099162 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f6e91d4b640 10 monclient(hunting): get_auth_request con 0x7f6e8c0b0160 auth_method 0 2024-08-09T14:50:50.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f6e91d4b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f6e91d4b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f6e91d4b640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f65cfc33640 10 monclient: shutdown 2024-08-09T14:50:50.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f65cfc33640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f65cfc33640 1 -- 172.21.15.155:0/451257900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f65c80b4ef0 msgr2=0x7f65c80b5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f65cfc33640 1 --2- 172.21.15.155:0/451257900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f65c80b4ef0 0x7f65c80b5760 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f65b80097a0 tx=0x7f65b802b960 comp rx=0 tx=0).stop 2024-08-09T14:50:50.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f65cfc33640 1 -- 172.21.15.155:0/451257900 shutdown_connections 2024-08-09T14:50:50.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f65cfc33640 1 --2- 172.21.15.155:0/451257900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f65c80b4ef0 0x7f65c80b5760 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.775 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.707+0000 7f65cfc33640 1 -- 172.21.15.155:0/451257900 >> 172.21.15.155:0/451257900 conn(0x7f65c806dbd0 msgr2=0x7f65c80676f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.708+0000 7f65cfc33640 1 -- 172.21.15.155:0/451257900 shutdown_connections 2024-08-09T14:50:50.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.708+0000 7f1b327fc640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b340b2d80 0x7f1b34145200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.776 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.708+0000 7f65cfc33640 1 -- 172.21.15.155:0/451257900 wait complete. 2024-08-09T14:50:50.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.708+0000 7f1b327fc640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b340b2d80 0x7f1b34145200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:47930/0 (socket says 172.21.15.155:47930) 2024-08-09T14:50:50.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.708+0000 7f1b327fc640 1 -- 172.21.15.155:0/3609027866 learned_addr learned my addr 172.21.15.155:0/3609027866 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.777 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.708+0000 7f1b327fc640 10 monclient(hunting): get_auth_request con 0x7f1b340b2d80 auth_method 0 2024-08-09T14:50:50.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.708+0000 7f1b327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.708+0000 7f1b327fc640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.778 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.708+0000 7f1b327fc640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.780 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.709+0000 7f65cfc33640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.780 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.709+0000 7f65cfc33640 10 monclient: monmap: 2024-08-09T14:50:50.780 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.780 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:50.781 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:50.781 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:50.781 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.781 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.781 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:50.781 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:50.781 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:50.781 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.781 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.709+0000 7f65cfc33640 1 librados: starting msgr at 2024-08-09T14:50:50.781 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.709+0000 7f65cfc33640 1 librados: starting objecter 2024-08-09T14:50:50.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.709+0000 7f65cfc33640 1 Processor -- start 2024-08-09T14:50:50.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.709+0000 7f65cfc33640 1 -- start start 2024-08-09T14:50:50.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.709+0000 7f65cfc33640 1 librados: setting wanted keys 2024-08-09T14:50:50.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.709+0000 7f65cfc33640 1 librados: calling monclient init 2024-08-09T14:50:50.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.709+0000 7f65cfc33640 10 monclient: init 2024-08-09T14:50:50.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.710+0000 7f65cfc33640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.710+0000 7f65cfc33640 10 monclient: _add_conns ranks=[1,0,2] 2024-08-09T14:50:50.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.710+0000 7f65cfc33640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f65c80b0160 0x7f65c812f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.782 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.710+0000 7f6e9154a640 1 --2- 172.21.15.155:0/364099162 >> v2:172.21.15.155:3300/0 conn(0x7f6e8c0b22e0 0x7f6e8c1469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.710+0000 7f65cfc33640 10 monclient(hunting): picked mon.noname-b con 0x7f65c80b0160 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:50.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.710+0000 7f65cfc33640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65c80b4ef0 0x7f65c812f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.710+0000 7f65cfc33640 10 monclient(hunting): picked mon.noname-a con 0x7f65c80b4ef0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:50.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.710+0000 7f65cfc33640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65c81444e0 0x7f65c81326b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.710+0000 7f65cfc33640 10 monclient(hunting): picked mon.noname-c con 0x7f65c81444e0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f65cfc33640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f65cfc33640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f65c8144970 con 0x7f65c80b4ef0 2024-08-09T14:50:50.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f6e91d4b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f6e91d4b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.785 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f6e91d4b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.785 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f65cfc33640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.785 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f65cfc33640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f65c80b2e70 con 0x7f65c80b0160 2024-08-09T14:50:50.785 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f65cfc33640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f65cfc33640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f65c8144af0 con 0x7f65c81444e0 2024-08-09T14:50:50.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f65cfc33640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f65cfc33640 10 monclient(hunting): authenticate will time out at 1257.748360s 2024-08-09T14:50:50.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f1b327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f1b327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.787 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f994972b640 1 --2- 172.21.15.155:0/2389289502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99441465f0 0x7f994413b530 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f993c007e60 tx=0x7f993c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.787 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f1b327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.787 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f99417fa640 10 client.?.objecter ms_handle_connect 0x7f99441465f0 2024-08-09T14:50:50.787 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f99417fa640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:50.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.711+0000 7f1b337fe640 1 --2- 172.21.15.155:0/3609027866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b340b4df0 0x7f1b34145740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.712+0000 7f6e9254c640 1 --2- 172.21.15.155:0/364099162 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e8c146f10 0x7f6e8c0b3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.712+0000 7f6e9154a640 10 monclient(hunting): get_auth_request con 0x7f6e8c0b22e0 auth_method 0 2024-08-09T14:50:50.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.712+0000 7f6e9154a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.712+0000 7f6e9154a640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.712+0000 7f6e9154a640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.712+0000 7f1b337fe640 10 monclient(hunting): get_auth_request con 0x7f1b340b4df0 auth_method 0 2024-08-09T14:50:50.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.712+0000 7f1b337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.712+0000 7f1b337fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.712+0000 7f1b337fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.712+0000 7f441551f640 1 --2- 172.21.15.155:0/2791730943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f441006bb80 0x7f4410070030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f440c0097a0 tx=0x7f440c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdf80fdef9d71739 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.713+0000 7fdabffff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fdab80c46d0 0x7fdab80c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.713+0000 7f6e9254c640 10 monclient(hunting): get_auth_request con 0x7f6e8c146f10 auth_method 0 2024-08-09T14:50:50.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.713+0000 7f6e9254c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.713+0000 7f6e9254c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.713+0000 7f6e9254c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.713+0000 7f6e9154a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.713+0000 7f6e9154a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.713+0000 7f6e9154a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.714+0000 7f1b32ffd640 1 --2- 172.21.15.155:0/3609027866 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b3406d8d0 0x7f1b340701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.715+0000 7f6e9254c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.715+0000 7f6e9254c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.715+0000 7f6e9254c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.716+0000 7f1b327fc640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2024-08-09T14:50:50.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.716+0000 7f1b327fc640 1 -- 172.21.15.155:0/3609027866 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b3406d8d0 msgr2=0x7f1b340701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.716+0000 7f1b327fc640 1 --2- 172.21.15.155:0/3609027866 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b3406d8d0 0x7f1b340701f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.716+0000 7f1b327fc640 1 -- 172.21.15.155:0/3609027866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b340b4df0 msgr2=0x7f1b34145740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.716+0000 7f1b327fc640 1 --2- 172.21.15.155:0/3609027866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b340b4df0 0x7f1b34145740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.716+0000 7f1b327fc640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.716+0000 7f1b327fc640 1 monclient: found mon.noname-b 2024-08-09T14:50:50.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.716+0000 7f1b327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.716+0000 7f1b327fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.716+0000 7f1b327fc640 1 -- 172.21.15.155:0/3609027866 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b34072e70 con 0x7f1b340b2d80 2024-08-09T14:50:50.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7f6e91d4b640 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2024-08-09T14:50:50.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7f6e91d4b640 1 -- 172.21.15.155:0/364099162 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e8c146f10 msgr2=0x7f6e8c0b3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7f6e91d4b640 1 --2- 172.21.15.155:0/364099162 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e8c146f10 0x7f6e8c0b3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7f6e91d4b640 1 -- 172.21.15.155:0/364099162 >> v2:172.21.15.155:3300/0 conn(0x7f6e8c0b22e0 msgr2=0x7f6e8c1469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7f6e91d4b640 1 --2- 172.21.15.155:0/364099162 >> v2:172.21.15.155:3300/0 conn(0x7f6e8c0b22e0 0x7f6e8c1469d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7f6e91d4b640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7f6e91d4b640 1 monclient: found mon.noname-c 2024-08-09T14:50:50.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7f6e91d4b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7f6e91d4b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7f6e91d4b640 1 -- 172.21.15.155:0/364099162 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e8c0b3d80 con 0x7f6e8c0b0160 2024-08-09T14:50:50.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7fdabffff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7fdab80c46d0 0x7fdab80c4af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:47946/0 (socket says 172.21.15.155:47946) 2024-08-09T14:50:50.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7fdabffff640 1 -- 172.21.15.155:0/3424337700 learned_addr learned my addr 172.21.15.155:0/3424337700 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7fdabffff640 10 monclient(hunting): get_auth_request con 0x7fdab80c46d0 auth_method 0 2024-08-09T14:50:50.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7fdabffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7fdabffff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.717+0000 7fdabffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.720+0000 7fdabffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.720+0000 7fdabffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.720+0000 7fdabffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.721+0000 7fdac4c59640 1 --2- 172.21.15.155:0/3424337700 >> v2:172.21.15.155:3300/0 conn(0x7fdab8039210 0x7fdab8035860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.721+0000 7f6e91d4b640 1 --2- 172.21.15.155:0/364099162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e8c0b0160 0x7f6e8c0b0540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c0b4610 tx=0x7f6e7c02b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e5798107824c3d2 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb397fff640 1 -- 172.21.15.155:0/4152612314 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a8043020 con 0x7fb3b013dc30 2024-08-09T14:50:50.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb397fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb397fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:50.797 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb397fff640 10 monclient: dump: 2024-08-09T14:50:50.797 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.797 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.797 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.797 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.797 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.797 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.798 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.798 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.798 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.798 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb397fff640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb397fff640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb397fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb397fff640 1 -- 172.21.15.155:0/4152612314 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3a8004250 con 0x7fb3b013dc30 2024-08-09T14:50:50.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb397fff640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb397fff640 1 -- 172.21.15.155:0/4152612314 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a8009510 con 0x7fb3b013dc30 2024-08-09T14:50:50.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb397fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.798 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb397fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:50.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb397fff640 10 monclient: dump: 2024-08-09T14:50:50.799 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.799 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.799 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.799 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.799 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.799 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.799 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.799 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.799 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.799 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb3b587b640 5 monclient: authenticate success, global_id 4338 2024-08-09T14:50:50.799 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb3b587b640 10 monclient: _renew_subs 2024-08-09T14:50:50.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb3b587b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb3b587b640 1 -- 172.21.15.155:0/4152612314 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3b0145230 con 0x7fb3b013dc30 2024-08-09T14:50:50.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb3ad7fa640 4 set_mon_vals no callback set 2024-08-09T14:50:50.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb3b587b640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb3b587b640 10 monclient: _renew_subs 2024-08-09T14:50:50.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb3b587b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb3b587b640 1 -- 172.21.15.155:0/4152612314 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3b0147060 con 0x7fb3b013dc30 2024-08-09T14:50:50.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.723+0000 7fb3b587b640 1 librados: init done 2024-08-09T14:50:50.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7fdac4c59640 10 monclient(hunting): get_auth_request con 0x7fdab8039210 auth_method 0 2024-08-09T14:50:50.800 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7fdac4c59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7fdac4c59640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7fdac4c59640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4407fff640 1 -- 172.21.15.155:0/2791730943 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c035020 con 0x7f441006bb80 2024-08-09T14:50:50.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4407fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4407fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:50.801 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4407fff640 10 monclient: dump: 2024-08-09T14:50:50.801 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.802 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.802 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.802 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.802 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.802 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.802 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.802 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.802 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.802 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.802 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4407fff640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.802 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4407fff640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.802 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4407fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4407fff640 1 -- 172.21.15.155:0/2791730943 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f440c031810 con 0x7f441006bb80 2024-08-09T14:50:50.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4407fff640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4407fff640 1 -- 172.21.15.155:0/2791730943 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f440c031be0 con 0x7f441006bb80 2024-08-09T14:50:50.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4407fff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4407fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:50.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4407fff640 10 monclient: dump: 2024-08-09T14:50:50.803 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.803 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.803 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.803 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.803 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.803 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.804 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.804 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.804 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.804 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4417783640 5 monclient: authenticate success, global_id 4295 2024-08-09T14:50:50.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4417783640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:50.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7f4417783640 4 set_mon_vals no callback set 2024-08-09T14:50:50.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7fb3b587b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7fb3b587b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.805 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7fb3b587b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.805 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.724+0000 7fb3b587b640 1 -- 172.21.15.155:0/4152612314 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3b00b0ab0 con 0x7fb3b013dc30 2024-08-09T14:50:50.805 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f65ce1d0640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65c81444e0 0x7f65c81326b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.805 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f4417783640 10 monclient: shutdown 2024-08-09T14:50:50.806 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f4417783640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.806 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f4417783640 1 -- 172.21.15.155:0/2791730943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f441006bb80 msgr2=0x7f4410070030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.806 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f4417783640 1 --2- 172.21.15.155:0/2791730943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f441006bb80 0x7f4410070030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f440c0097a0 tx=0x7f440c02b8b0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.806 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f65ce1d0640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65c81444e0 0x7f65c81326b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54756/0 (socket says 172.21.15.155:54756) 2024-08-09T14:50:50.806 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f65ce1d0640 1 -- 172.21.15.155:0/1311888196 learned_addr learned my addr 172.21.15.155:0/1311888196 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.806 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f65ce1d0640 10 monclient(hunting): get_auth_request con 0x7f65c81444e0 auth_method 0 2024-08-09T14:50:50.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f4417783640 1 -- 172.21.15.155:0/2791730943 shutdown_connections 2024-08-09T14:50:50.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f65ce1d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f65ce1d0640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f65ce1d0640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7fdac4c59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7fdac4c59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7fdac4c59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.808 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f4417783640 1 --2- 172.21.15.155:0/2791730943 >> v2:172.21.15.194:3300/0 conn(0x7f4410072fa0 0x7f44100733c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.808 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f4417783640 1 --2- 172.21.15.155:0/2791730943 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f441006bb80 0x7f4410070030 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f4417783640 1 -- 172.21.15.155:0/2791730943 >> 172.21.15.155:0/2791730943 conn(0x7f44100b5af0 msgr2=0x7f44100b5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f4417783640 1 -- 172.21.15.155:0/2791730943 shutdown_connections 2024-08-09T14:50:50.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.725+0000 7f4417783640 1 -- 172.21.15.155:0/2791730943 wait complete. 2024-08-09T14:50:50.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7fdabffff640 10 monclient(hunting): handle_auth_done global_id 4321 payload 274 2024-08-09T14:50:50.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7fdabffff640 1 -- 172.21.15.155:0/3424337700 >> v2:172.21.15.155:3300/0 conn(0x7fdab8039210 msgr2=0x7fdab8035860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7fdabffff640 1 --2- 172.21.15.155:0/3424337700 >> v2:172.21.15.155:3300/0 conn(0x7fdab8039210 0x7fdab8035860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7fdabffff640 1 -- 172.21.15.155:0/3424337700 >> v1:172.21.15.194:6789/0 conn(0x7fdab80cac70 legacy=0x7fdab80cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:50.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7fdabffff640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7fdabffff640 1 monclient: found mon.noname-c 2024-08-09T14:50:50.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7fdabffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7fdabffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7fdabffff640 1 -- 172.21.15.155:0/3424337700 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab8036170 con 0x7fdab80c46d0 2024-08-09T14:50:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7f4417783640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7f4417783640 10 monclient: monmap: 2024-08-09T14:50:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:50.812 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7f4417783640 1 librados: starting msgr at 2024-08-09T14:50:50.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7f4417783640 1 librados: starting objecter 2024-08-09T14:50:50.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7f4417783640 1 Processor -- start 2024-08-09T14:50:50.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7f4417783640 1 -- start start 2024-08-09T14:50:50.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7f4417783640 1 librados: setting wanted keys 2024-08-09T14:50:50.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7f4417783640 1 librados: calling monclient init 2024-08-09T14:50:50.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.726+0000 7f4417783640 10 monclient: init 2024-08-09T14:50:50.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f1b327fc640 1 --2- 172.21.15.155:0/3609027866 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b340b2d80 0x7f1b34145200 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009890 tx=0x7f1b280046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f1b1b7fe640 10 client.?.objecter ms_handle_connect 0x7f1b340b2d80 2024-08-09T14:50:50.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f1b1b7fe640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:50.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f65ce1d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f65ce1d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f65ce1d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e90d49640 1 -- 172.21.15.155:0/364099162 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c038070 con 0x7f6e8c0b0160 2024-08-09T14:50:50.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e90d49640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e90d49640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:50.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e90d49640 10 monclient: dump: 2024-08-09T14:50:50.815 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.815 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.815 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.815 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.815 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.815 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.815 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.815 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.815 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.816 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e90d49640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e90d49640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e90d49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e90d49640 1 -- 172.21.15.155:0/364099162 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e7c031740 con 0x7f6e8c0b0160 2024-08-09T14:50:50.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e90d49640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e90d49640 1 -- 172.21.15.155:0/364099162 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c02f050 con 0x7f6e8c0b0160 2024-08-09T14:50:50.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e90d49640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e90d49640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:50.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e90d49640 10 monclient: dump: 2024-08-09T14:50:50.816 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.816 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.816 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.817 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.817 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.817 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.817 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.817 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.817 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.817 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e93faf640 5 monclient: authenticate success, global_id 4315 2024-08-09T14:50:50.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e93faf640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:50.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f6e93faf640 4 set_mon_vals no callback set 2024-08-09T14:50:50.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 10 monclient: _add_conns ranks=[1,0,2] 2024-08-09T14:50:50.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f441006bb80 0x7f44100b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 10 monclient(hunting): picked mon.noname-b con 0x7f441006bb80 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:50.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4410072fa0 0x7f44101429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 10 monclient(hunting): picked mon.noname-a con 0x7f4410072fa0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44101470c0 0x7f44101dd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 10 monclient(hunting): picked mon.noname-c con 0x7f44101470c0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4410147c20 con 0x7f4410072fa0 2024-08-09T14:50:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4410073900 con 0x7f441006bb80 2024-08-09T14:50:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f4410147da0 con 0x7f44101470c0 2024-08-09T14:50:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.727+0000 7f4417783640 10 monclient(hunting): authenticate will time out at 1257.764876s 2024-08-09T14:50:50.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.728+0000 7f65cd1ce640 1 --2- 172.21.15.155:0/1311888196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65c80b4ef0 0x7f65c812f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.728+0000 7f6e93faf640 10 monclient: shutdown 2024-08-09T14:50:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.728+0000 7f6e93faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.728+0000 7f6e93faf640 1 -- 172.21.15.155:0/364099162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e8c0b0160 msgr2=0x7f6e8c0b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.728+0000 7f6e93faf640 1 --2- 172.21.15.155:0/364099162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e8c0b0160 0x7f6e8c0b0540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c0b4610 tx=0x7f6e7c02b7e0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.728+0000 7f6e93faf640 1 -- 172.21.15.155:0/364099162 shutdown_connections 2024-08-09T14:50:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.728+0000 7f6e93faf640 1 --2- 172.21.15.155:0/364099162 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e8c146f10 0x7f6e8c0b3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.728+0000 7f6e93faf640 1 --2- 172.21.15.155:0/364099162 >> v2:172.21.15.155:3300/0 conn(0x7f6e8c0b22e0 0x7f6e8c1469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.728+0000 7f6e93faf640 1 --2- 172.21.15.155:0/364099162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e8c0b0160 0x7f6e8c0b0540 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.728+0000 7f6e93faf640 1 -- 172.21.15.155:0/364099162 >> 172.21.15.155:0/364099162 conn(0x7f6e8c06dbd0 msgr2=0x7f6e8c0676f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:50.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.728+0000 7f6e93faf640 1 -- 172.21.15.155:0/364099162 shutdown_connections 2024-08-09T14:50:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.728+0000 7f6e93faf640 1 -- 172.21.15.155:0/364099162 wait complete. 2024-08-09T14:50:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.729+0000 7f6e93faf640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.729+0000 7f6e93faf640 10 monclient: monmap: 2024-08-09T14:50:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:50.822 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:50.822 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.729+0000 7f6e93faf640 1 librados: starting msgr at 2024-08-09T14:50:50.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.729+0000 7f6e93faf640 1 librados: starting objecter 2024-08-09T14:50:50.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.729+0000 7f6e93faf640 1 Processor -- start 2024-08-09T14:50:50.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.729+0000 7f6e93faf640 1 -- start start 2024-08-09T14:50:50.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.729+0000 7f6e93faf640 1 librados: setting wanted keys 2024-08-09T14:50:50.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.729+0000 7f6e93faf640 1 librados: calling monclient init 2024-08-09T14:50:50.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.729+0000 7f6e93faf640 10 monclient: init 2024-08-09T14:50:50.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 10 monclient: _add_conns ranks=[0,2,1] 2024-08-09T14:50:50.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e8c0b0160 0x7f6e8c072750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 10 monclient(hunting): picked mon.noname-a con 0x7f6e8c0b0160 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:50.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e8c0b22e0 0x7f6e8c06fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 10 monclient(hunting): picked mon.noname-c con 0x7f6e8c0b22e0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e8c146f10 0x7f6e8c070340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 10 monclient(hunting): picked mon.noname-b con 0x7f6e8c146f10 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:50.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8c142cc0 con 0x7f6e8c0b0160 2024-08-09T14:50:50.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8c0b4310 con 0x7f6e8c146f10 2024-08-09T14:50:50.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f6e8c142e40 con 0x7f6e8c0b22e0 2024-08-09T14:50:50.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f6e93faf640 10 monclient(hunting): authenticate will time out at 1257.767385s 2024-08-09T14:50:50.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.730+0000 7f65cd9cf640 1 --2- 172.21.15.155:0/1311888196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f65c80b0160 0x7f65c812f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.731+0000 7f65cd1ce640 10 monclient(hunting): get_auth_request con 0x7f65c80b4ef0 auth_method 0 2024-08-09T14:50:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.731+0000 7f65cd1ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.731+0000 7f65cd1ce640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.731+0000 7f65cd1ce640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.733+0000 7f65cd9cf640 10 monclient(hunting): get_auth_request con 0x7f65c80b0160 auth_method 0 2024-08-09T14:50:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.733+0000 7f65cd9cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.733+0000 7f65cd9cf640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.733+0000 7f65cd9cf640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.733+0000 7f65ce1d0640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-08-09T14:50:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.733+0000 7f65ce1d0640 1 -- 172.21.15.155:0/1311888196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f65c80b0160 msgr2=0x7f65c812f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.733+0000 7f65ce1d0640 1 --2- 172.21.15.155:0/1311888196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f65c80b0160 0x7f65c812f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.733+0000 7f65ce1d0640 1 -- 172.21.15.155:0/1311888196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65c80b4ef0 msgr2=0x7f65c812f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.733+0000 7f65ce1d0640 1 --2- 172.21.15.155:0/1311888196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65c80b4ef0 0x7f65c812f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.733+0000 7f65ce1d0640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.733+0000 7f65ce1d0640 1 monclient: found mon.noname-c 2024-08-09T14:50:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.733+0000 7f65ce1d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.733+0000 7f65ce1d0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.733+0000 7f65ce1d0640 1 -- 172.21.15.155:0/1311888196 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c80aaf10 con 0x7f65c81444e0 2024-08-09T14:50:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b1b7fe640 1 -- 172.21.15.155:0/3609027866 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b28035040 con 0x7f1b340b2d80 2024-08-09T14:50:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b1b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:50.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b1b7fe640 10 monclient: dump: 2024-08-09T14:50:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.828 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b1b7fe640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b1b7fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f441551f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f441006bb80 0x7f44100b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b1b7fe640 1 -- 172.21.15.155:0/3609027866 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b2803b900 con 0x7f1b340b2d80 2024-08-09T14:50:50.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b1b7fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b317fa640 4 set_mon_vals no callback set 2024-08-09T14:50:50.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b3937e640 5 monclient: authenticate success, global_id 4318 2024-08-09T14:50:50.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b1b7fe640 1 -- 172.21.15.155:0/3609027866 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2803bcf0 con 0x7f1b340b2d80 2024-08-09T14:50:50.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b3937e640 10 monclient: _renew_subs 2024-08-09T14:50:50.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b3937e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b3937e640 1 -- 172.21.15.155:0/3609027866 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b340717f0 con 0x7f1b340b2d80 2024-08-09T14:50:50.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b1b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:50.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b1b7fe640 10 monclient: dump: 2024-08-09T14:50:50.831 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.831 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.831 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.831 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.831 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.831 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.831 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.831 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.831 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.831 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b3937e640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b3937e640 10 monclient: _renew_subs 2024-08-09T14:50:50.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b3937e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b3937e640 1 -- 172.21.15.155:0/3609027866 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b34071db0 con 0x7f1b340b2d80 2024-08-09T14:50:50.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f1b3937e640 1 librados: init done 2024-08-09T14:50:50.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.734+0000 7f441551f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f441006bb80 0x7f44100b3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:47958/0 (socket says 172.21.15.155:47958) 2024-08-09T14:50:50.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.735+0000 7f441551f640 1 -- 172.21.15.155:0/1336843718 learned_addr learned my addr 172.21.15.155:0/1336843718 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.735+0000 7f441551f640 10 monclient(hunting): get_auth_request con 0x7f441006bb80 auth_method 0 2024-08-09T14:50:50.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.735+0000 7f441551f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.735+0000 7f441551f640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.735+0000 7f441551f640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.736+0000 7f1b3937e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.736+0000 7f1b3937e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.736+0000 7f1b3937e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:50.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.736+0000 7f65ce1d0640 1 --2- 172.21.15.155:0/1311888196 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65c81444e0 0x7f65c81326b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65c000c730 tx=0x7f65c000cb30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.736+0000 7f65be7fc640 10 client.?.objecter ms_handle_connect 0x7f65c81444e0 2024-08-09T14:50:50.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.736+0000 7f65be7fc640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:50.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.736+0000 7f1b3937e640 1 -- 172.21.15.155:0/3609027866 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b34073810 con 0x7f1b340b2d80 2024-08-09T14:50:50.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.737+0000 7f441551f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.737+0000 7f441551f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.737+0000 7f441551f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.740+0000 7f4415d20640 1 --2- 172.21.15.155:0/1336843718 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44101470c0 0x7f44101dd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.740+0000 7f4415d20640 10 monclient(hunting): get_auth_request con 0x7f44101470c0 auth_method 0 2024-08-09T14:50:50.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.740+0000 7f4415d20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.740+0000 7f4415d20640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.740+0000 7f4415d20640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.741+0000 7f4415d20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.741+0000 7f4415d20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.741+0000 7f4415d20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.741+0000 7f6e9154a640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e8c0b22e0 0x7f6e8c06fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.741+0000 7f65be7fc640 1 -- 172.21.15.155:0/1311888196 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c0016020 con 0x7f65c81444e0 2024-08-09T14:50:50.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.741+0000 7f65be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.741+0000 7f65be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:50.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.741+0000 7f65be7fc640 10 monclient: dump: 2024-08-09T14:50:50.838 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.838 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.838 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.838 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.838 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.838 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.838 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.838 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.838 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.838 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65be7fc640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65be7fc640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65be7fc640 1 -- 172.21.15.155:0/1311888196 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65c0006d50 con 0x7f65c81444e0 2024-08-09T14:50:50.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65be7fc640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65be7fc640 1 -- 172.21.15.155:0/1311888196 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c0003bb0 con 0x7f65c81444e0 2024-08-09T14:50:50.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65be7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:50.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65be7fc640 10 monclient: dump: 2024-08-09T14:50:50.839 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.839 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.839 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.839 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.839 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.839 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.840 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.840 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.840 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.840 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65cfc33640 5 monclient: authenticate success, global_id 4304 2024-08-09T14:50:50.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65cfc33640 10 monclient: _renew_subs 2024-08-09T14:50:50.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65cfc33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65cfc33640 1 -- 172.21.15.155:0/1311888196 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65c8071460 con 0x7f65c81444e0 2024-08-09T14:50:50.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65bffff640 4 set_mon_vals no callback set 2024-08-09T14:50:50.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65cfc33640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65cfc33640 10 monclient: _renew_subs 2024-08-09T14:50:50.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65cfc33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65cfc33640 1 -- 172.21.15.155:0/1311888196 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f65c8132dd0 con 0x7f65c81444e0 2024-08-09T14:50:50.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f65cfc33640 1 librados: init done 2024-08-09T14:50:50.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f6e9154a640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e8c0b22e0 0x7f6e8c06fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54772/0 (socket says 172.21.15.155:54772) 2024-08-09T14:50:50.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f6e9154a640 1 -- 172.21.15.155:0/1044154739 learned_addr learned my addr 172.21.15.155:0/1044154739 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:50.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f6e9154a640 10 monclient(hunting): get_auth_request con 0x7f6e8c0b22e0 auth_method 0 2024-08-09T14:50:50.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f6e9154a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:50.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f6e9154a640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:50.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.742+0000 7f6e9154a640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:50.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.743+0000 7f65cfc33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.743+0000 7f65cfc33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.743+0000 7f65cfc33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.743+0000 7f65cfc33640 1 -- 172.21.15.155:0/1311888196 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65c80b5dd0 con 0x7f65c81444e0 2024-08-09T14:50:50.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.746+0000 7f4415d20640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-08-09T14:50:50.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.746+0000 7f4415d20640 1 -- 172.21.15.155:0/1336843718 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f441006bb80 msgr2=0x7f44100b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.746+0000 7f4415d20640 1 --2- 172.21.15.155:0/1336843718 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f441006bb80 0x7f44100b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.746+0000 7f4415d20640 1 -- 172.21.15.155:0/1336843718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4410072fa0 msgr2=0x7f44101429d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:50.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.746+0000 7f4415d20640 1 --2- 172.21.15.155:0/1336843718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4410072fa0 0x7f44101429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:50.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.746+0000 7f4415d20640 10 monclient: _finish_hunting 0 2024-08-09T14:50:50.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.746+0000 7f4415d20640 1 monclient: found mon.noname-c 2024-08-09T14:50:50.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.746+0000 7f4415d20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:50.844 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.746+0000 7f4415d20640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:50:50.844 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.746+0000 7f4415d20640 1 -- 172.21.15.155:0/1336843718 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44101dcb90 con 0x7f44101470c0 2024-08-09T14:50:50.844 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.749+0000 7f4415d20640 1 --2- 172.21.15.155:0/1336843718 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44101470c0 0x7f44101dd0b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f440800caa0 tx=0x7f440800cf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.844 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.749+0000 7f4405ffb640 10 client.?.objecter ms_handle_connect 0x7f44101470c0 2024-08-09T14:50:50.844 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.749+0000 7f4405ffb640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:50.844 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.750+0000 7f6e9154a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:50.844 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.750+0000 7f6e9154a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:50.844 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.750+0000 7f6e9154a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:50.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.752+0000 7f191bbb8640 10 monclient: get_monmap_and_config 2024-08-09T14:50:50.845 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.752+0000 7f191bbb8640 10 monclient: build_initial_monmap 2024-08-09T14:50:50.846 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.752+0000 7f191bbb8640 10 monclient: monmap: 2024-08-09T14:50:50.846 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:50.846 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.846 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:50.753148+0000 2024-08-09T14:50:50.846 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:50.753148+0000 2024-08-09T14:50:50.846 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:50.846 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.846 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:50.846 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:50.846 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:50.846 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:50.846 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:50.846 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.752+0000 7f191bbb8640 1 Processor -- start 2024-08-09T14:50:50.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.752+0000 7f191bbb8640 1 -- start start 2024-08-09T14:50:50.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.752+0000 7f191bbb8640 10 monclient: init 2024-08-09T14:50:50.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.753+0000 7f191bbb8640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:50.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.753+0000 7f191bbb8640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-08-09T14:50:50.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.753+0000 7f191bbb8640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f19140b0160 0x7f19140b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.753+0000 7f191bbb8640 10 monclient(hunting): picked mon.noname-c con 0x7f19140b0160 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:50.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.753+0000 7f191bbb8640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19140b22e0 0x7f19141469d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.753+0000 7f191bbb8640 10 monclient(hunting): picked mon.noname-b con 0x7f19140b22e0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:50.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.753+0000 7f191bbb8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1914146f10 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:50.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.753+0000 7f191bbb8640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19141461c0 con 0x7f1914146f10 2024-08-09T14:50:50.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.753+0000 7f191bbb8640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.753+0000 7f191bbb8640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f19140b26e0 con 0x7f19140b0160 2024-08-09T14:50:50.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.753+0000 7f191bbb8640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:50.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.753+0000 7f191bbb8640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f19140b2860 con 0x7f19140b22e0 2024-08-09T14:50:50.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.753+0000 7f191bbb8640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:50.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.753+0000 7f191bbb8640 10 monclient(hunting): authenticate will time out at 1257.790548s 2024-08-09T14:50:50.938 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f99417fa640 1 -- 172.21.15.155:0/2389289502 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c014420 con 0x7f99441465f0 2024-08-09T14:50:50.938 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f99417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:50.938 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f99417fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:50.938 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f99417fa640 10 monclient: dump: 2024-08-09T14:50:50.938 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:50.938 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:50.938 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.938 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:50.938 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:50.939 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:50.939 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:50.939 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:50.939 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:50.939 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:50.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f99417fa640 10 monclient: _finish_auth 0 2024-08-09T14:50:50.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f99417fa640 10 monclient: _check_auth_tickets 2024-08-09T14:50:50.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f99417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:50.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f99417fa640 1 -- 172.21.15.155:0/2389289502 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f993c019070 con 0x7f99441465f0 2024-08-09T14:50:50.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f9949f2c640 5 monclient: authenticate success, global_id 4341 2024-08-09T14:50:50.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f9949f2c640 10 monclient: _renew_subs 2024-08-09T14:50:50.939 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f9949f2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.940 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f9949f2c640 1 -- 172.21.15.155:0/2389289502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99441cccf0 con 0x7f99441465f0 2024-08-09T14:50:50.940 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f99417fa640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:50.940 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f9942ffd640 4 set_mon_vals no callback set 2024-08-09T14:50:50.940 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f9949f2c640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:50.940 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f9949f2c640 10 monclient: _renew_subs 2024-08-09T14:50:50.940 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f9949f2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.940 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f9949f2c640 1 -- 172.21.15.155:0/2389289502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99441cd2b0 con 0x7f99441465f0 2024-08-09T14:50:50.940 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.936+0000 7f9949f2c640 1 librados: init done 2024-08-09T14:50:50.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.937+0000 7f9949f2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.937+0000 7f9949f2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:50.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.937+0000 7f9949f2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:50.941 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.937+0000 7f9949f2c640 1 -- 172.21.15.155:0/2389289502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99440b0b30 con 0x7f99441465f0 2024-08-09T14:50:50.984 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.983+0000 7f1b1b7fe640 1 -- 172.21.15.155:0/3609027866 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1b28038020 con 0x7f1b340b2d80 2024-08-09T14:50:50.984 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.983+0000 7f1b1b7fe640 1 --2- 172.21.15.155:0/3609027866 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f1b2006d7c0 0x7f1b2006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:50.985 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.983+0000 7f1b32ffd640 1 --2- 172.21.15.155:0/3609027866 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f1b2006d7c0 0x7f1b2006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:50.985 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.983+0000 7f1b32ffd640 10 monclient: get_auth_request con 0x7f1b2006d7c0 auth_method 0 2024-08-09T14:50:50.985 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.984+0000 7f1b1b7fe640 10 client.4318.objecter ms_handle_connect 0x7f1b2006d7c0 2024-08-09T14:50:50.985 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.983+0000 7f1b32ffd640 1 --2- 172.21.15.155:0/3609027866 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f1b2006d7c0 0x7f1b2006fc80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0012e0 tx=0x7f1b2c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:50.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.992+0000 7f1b1b7fe640 1 -- 172.21.15.155:0/3609027866 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f1b28079600 con 0x7f1b340b2d80 2024-08-09T14:50:50.993 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.992+0000 7f1b1b7fe640 10 client.4318.objecter ms_dispatch 0x7f1b34000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:50.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.992+0000 7f1b1b7fe640 3 client.4318.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:50.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.992+0000 7f1b1b7fe640 3 client.4318.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:50.994 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:50.992+0000 7f1b1b7fe640 20 client.4318.objecter dump_active .. 0 homeless 2024-08-09T14:50:51.003 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.002+0000 7f1b1b7fe640 1 -- 172.21.15.155:0/3609027866 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f1b34073810 con 0x7f1b340b2d80 2024-08-09T14:50:51.004 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.002+0000 7f1b1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.004 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.002+0000 7f1b1b7fe640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:51.006 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.005+0000 7f6e9254c640 1 --2- 172.21.15.155:0/1044154739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e8c146f10 0x7f6e8c070340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:51.008 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.007+0000 7f99417fa640 1 -- 172.21.15.155:0/2389289502 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993c0147c0 con 0x7f99441465f0 2024-08-09T14:50:51.008 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.007+0000 7f99417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:51.008 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.007+0000 7f99417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:51.008 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.007+0000 7f99417fa640 10 monclient: dump: 2024-08-09T14:50:51.008 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:51.008 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:51.008 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.009 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.009 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:51.009 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.009 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:51.009 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:51.009 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:51.009 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.007+0000 7fb397fff640 1 -- 172.21.15.155:0/4152612314 <== mon.0 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb3a8005d10 con 0x7fb3b013dc30 2024-08-09T14:50:51.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.007+0000 7fb397fff640 1 --2- 172.21.15.155:0/4152612314 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fb39c06d810 0x7fb39c06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:51.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.008+0000 7fb3aeffd640 1 --2- 172.21.15.155:0/4152612314 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fb39c06d810 0x7fb39c06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:51.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.008+0000 7fb3aeffd640 10 monclient: get_auth_request con 0x7fb39c06d810 auth_method 0 2024-08-09T14:50:51.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.008+0000 7fb397fff640 10 client.4338.objecter ms_handle_connect 0x7fb39c06d810 2024-08-09T14:50:51.010 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.008+0000 7fb3aeffd640 1 --2- 172.21.15.155:0/4152612314 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fb39c06d810 0x7fb39c06fcd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0006a90 tx=0x7fb3a0006a20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:51.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.013+0000 7f6e9154a640 10 monclient(hunting): handle_auth_done global_id 4310 payload 824 2024-08-09T14:50:51.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.013+0000 7f6e9154a640 1 -- 172.21.15.155:0/1044154739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e8c146f10 msgr2=0x7f6e8c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.013+0000 7f6e9154a640 1 --2- 172.21.15.155:0/1044154739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e8c146f10 0x7f6e8c070340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.013+0000 7f6e9154a640 1 -- 172.21.15.155:0/1044154739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e8c0b0160 msgr2=0x7f6e8c072750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.013+0000 7f6e9154a640 1 --2- 172.21.15.155:0/1044154739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e8c0b0160 0x7f6e8c072750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.013+0000 7f6e9154a640 10 monclient: _finish_hunting 0 2024-08-09T14:50:51.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.013+0000 7f6e9154a640 1 monclient: found mon.noname-c 2024-08-09T14:50:51.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.013+0000 7f6e9154a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:51.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.013+0000 7f6e9154a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:50:51.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.013+0000 7f6e9154a640 1 -- 172.21.15.155:0/1044154739 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e8c0716c0 con 0x7f6e8c0b22e0 2024-08-09T14:50:51.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.018+0000 7f1919954640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f19140b0160 0x7f19140b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:51.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.022+0000 7fdabffff640 1 --2- 172.21.15.155:0/3424337700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdab80c46d0 0x7fdab80c4af0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdab8008050 tx=0x7fdaac02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=250b2fe96ec0ff4b server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:51.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4405ffb640 1 -- 172.21.15.155:0/1336843718 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408017070 con 0x7f44101470c0 2024-08-09T14:50:51.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4405ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:51.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4405ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:51.029 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4405ffb640 10 monclient: dump: 2024-08-09T14:50:51.030 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:51.030 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:51.030 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.030 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.030 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:51.030 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.030 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:51.030 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:51.030 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:51.030 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4405ffb640 10 monclient: _finish_auth 0 2024-08-09T14:50:51.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4405ffb640 10 monclient: _check_auth_tickets 2024-08-09T14:50:51.030 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4405ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:51.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4405ffb640 1 -- 172.21.15.155:0/1336843718 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f440800aac0 con 0x7f44101470c0 2024-08-09T14:50:51.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4417783640 5 monclient: authenticate success, global_id 4307 2024-08-09T14:50:51.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4417783640 10 monclient: _renew_subs 2024-08-09T14:50:51.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4417783640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:51.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4417783640 1 -- 172.21.15.155:0/1336843718 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44101dd5f0 con 0x7f44101470c0 2024-08-09T14:50:51.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4405ffb640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:51.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4405ffb640 1 -- 172.21.15.155:0/1336843718 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4408012400 con 0x7f44101470c0 2024-08-09T14:50:51.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4405ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:51.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4405ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:51.031 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4405ffb640 10 monclient: dump: 2024-08-09T14:50:51.031 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:51.031 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:51.031 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.032 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.032 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:51.032 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.032 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:51.032 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:51.032 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:51.032 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.032 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f65be7fc640 1 -- 172.21.15.155:0/1311888196 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f65c00050d0 con 0x7f65c81444e0 2024-08-09T14:50:51.033 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4417783640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:51.033 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4417783640 10 monclient: _renew_subs 2024-08-09T14:50:51.033 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f44077fe640 4 set_mon_vals no callback set 2024-08-09T14:50:51.033 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4417783640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:51.033 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4417783640 1 -- 172.21.15.155:0/1336843718 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f44100b4840 con 0x7f44101470c0 2024-08-09T14:50:51.033 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f4417783640 1 librados: init done 2024-08-09T14:50:51.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.028+0000 7f65be7fc640 1 --2- 172.21.15.155:0/1311888196 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f65ac06d7f0 0x7f65ac06fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:51.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f65cd9cf640 1 --2- 172.21.15.155:0/1311888196 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f65ac06d7f0 0x7f65ac06fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:51.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f65cd9cf640 10 monclient: get_auth_request con 0x7f65ac06d7f0 auth_method 0 2024-08-09T14:50:51.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f1919954640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f19140b0160 0x7f19140b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:47978/0 (socket says 172.21.15.155:47978) 2024-08-09T14:50:51.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f1919954640 1 -- 172.21.15.155:0/1728482897 learned_addr learned my addr 172.21.15.155:0/1728482897 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:51.034 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f1919954640 10 monclient(hunting): get_auth_request con 0x7f19140b0160 auth_method 0 2024-08-09T14:50:51.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f1919954640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:51.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f1919954640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:51.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f1919954640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:51.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f65be7fc640 10 client.4304.objecter ms_handle_connect 0x7f65ac06d7f0 2024-08-09T14:50:51.035 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f65cd9cf640 1 --2- 172.21.15.155:0/1311888196 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f65ac06d7f0 0x7f65ac06fcb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f65c4005880 tx=0x7f65c4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:51.036 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f4417783640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.036 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f4417783640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.036 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f4417783640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:51.036 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f4417783640 1 -- 172.21.15.155:0/1336843718 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4410148b00 con 0x7f44101470c0 2024-08-09T14:50:51.036 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f1919954640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:51.036 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f1919954640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:51.037 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.029+0000 7f1919954640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:51.038 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdabf7fe640 1 -- 172.21.15.155:0/3424337700 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac034070 con 0x7fdab80c46d0 2024-08-09T14:50:51.038 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdabf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:51.038 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdabf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:51.038 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdabf7fe640 10 monclient: dump: 2024-08-09T14:50:51.038 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:51.038 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:51.038 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.038 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.038 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:51.038 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.039 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:51.039 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:51.039 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:51.039 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdabf7fe640 10 monclient: _finish_auth 0 2024-08-09T14:50:51.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdabf7fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:51.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdabf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:51.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdabf7fe640 1 -- 172.21.15.155:0/3424337700 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaac030860 con 0x7fdab80c46d0 2024-08-09T14:50:51.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdabf7fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:51.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdabf7fe640 1 -- 172.21.15.155:0/3424337700 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaac030c30 con 0x7fdab80c46d0 2024-08-09T14:50:51.039 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdac5c5b640 5 monclient: authenticate success, global_id 4321 2024-08-09T14:50:51.040 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdac5c5b640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:51.040 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdac5c5b640 4 set_mon_vals no callback set 2024-08-09T14:50:51.040 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdabf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:51.040 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdabf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:51.040 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.031+0000 7fdabf7fe640 10 monclient: dump: 2024-08-09T14:50:51.040 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:51.040 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:51.040 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.040 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.040 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:51.040 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.040 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:51.040 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:51.041 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:51.041 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7fdac5c5b640 10 monclient: shutdown 2024-08-09T14:50:51.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7fdac5c5b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:51.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7fdac5c5b640 1 -- 172.21.15.155:0/3424337700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdab80c46d0 msgr2=0x7fdab80c4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7fdac5c5b640 1 --2- 172.21.15.155:0/3424337700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdab80c46d0 0x7fdab80c4af0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdab8008050 tx=0x7fdaac02b9a0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7fdac5c5b640 1 -- 172.21.15.155:0/3424337700 shutdown_connections 2024-08-09T14:50:51.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7fdac5c5b640 1 --2- 172.21.15.155:0/3424337700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdab80c46d0 0x7fdab80c4af0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7fdac5c5b640 1 --2- 172.21.15.155:0/3424337700 >> v2:172.21.15.155:3300/0 conn(0x7fdab8039210 0x7fdab8035860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7fdac5c5b640 1 -- 172.21.15.155:0/3424337700 >> 172.21.15.155:0/3424337700 conn(0x7fdab8006c00 msgr2=0x7fdab8037880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:51.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7fdac5c5b640 1 -- 172.21.15.155:0/3424337700 shutdown_connections 2024-08-09T14:50:51.041 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7fdac5c5b640 1 -- 172.21.15.155:0/3424337700 wait complete. 2024-08-09T14:50:51.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7f1919954640 10 monclient(hunting): handle_auth_done global_id 4330 payload 274 2024-08-09T14:50:51.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7f1919954640 1 -- 172.21.15.155:0/1728482897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19140b22e0 msgr2=0x7f19141469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:51.042 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7f1919954640 1 --2- 172.21.15.155:0/1728482897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19140b22e0 0x7f19141469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.043 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7f1919954640 1 -- 172.21.15.155:0/1728482897 >> v1:172.21.15.155:6789/0 conn(0x7f1914146f10 legacy=0x7f19140b3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:51.043 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7f1919954640 10 monclient: _finish_hunting 0 2024-08-09T14:50:51.043 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7f1919954640 1 monclient: found mon.noname-c 2024-08-09T14:50:51.043 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7f1919954640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:51.043 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7f1919954640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:50:51.043 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.032+0000 7f1919954640 1 -- 172.21.15.155:0/1728482897 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19140b3f50 con 0x7f19140b0160 2024-08-09T14:50:51.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.033+0000 7fdac5c5b640 10 monclient: build_initial_monmap 2024-08-09T14:50:51.044 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.033+0000 7fdac5c5b640 10 monclient: monmap: 2024-08-09T14:50:51.044 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:51.044 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:51.044 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:51.045 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:51.045 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:51.045 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.045 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:51.045 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:51.045 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:51.045 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.033+0000 7fdac5c5b640 1 librados: starting msgr at 2024-08-09T14:50:51.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.033+0000 7fdac5c5b640 1 librados: starting objecter 2024-08-09T14:50:51.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.033+0000 7fdac5c5b640 1 Processor -- start 2024-08-09T14:50:51.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.033+0000 7fdac5c5b640 1 -- start start 2024-08-09T14:50:51.045 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.033+0000 7fdac5c5b640 1 librados: setting wanted keys 2024-08-09T14:50:51.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.033+0000 7fdac5c5b640 1 librados: calling monclient init 2024-08-09T14:50:51.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.033+0000 7fdac5c5b640 10 monclient: init 2024-08-09T14:50:51.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:51.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 10 monclient: _add_conns ranks=[2,0,1] 2024-08-09T14:50:51.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdab8039210 0x7fdab802fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:51.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 10 monclient(hunting): picked mon.noname-c con 0x7fdab8039210 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:51.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdab80c46d0 0x7fdab80c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:51.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 10 monclient(hunting): picked mon.noname-a con 0x7fdab80c46d0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:51.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdab80cac70 0x7fdab8160a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:51.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 10 monclient(hunting): picked mon.noname-b con 0x7fdab80cac70 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:51.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:51.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fdab80cb960 con 0x7fdab80c46d0 2024-08-09T14:50:51.046 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:51.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdab80cb7e0 con 0x7fdab80cac70 2024-08-09T14:50:51.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:51.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fdab80d1f70 con 0x7fdab8039210 2024-08-09T14:50:51.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:51.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.034+0000 7fdac5c5b640 10 monclient(hunting): authenticate will time out at 1258.071601s 2024-08-09T14:50:51.047 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.036+0000 7f1919954640 1 --2- 172.21.15.155:0/1728482897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19140b0160 0x7f19140b0540 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1910009870 tx=0x7f191002b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9d3f9188fc045f9 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:51.048 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.037+0000 7fb397fff640 1 -- 172.21.15.155:0/4152612314 <== mon.0 v2:172.21.15.155:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7fb3a8079e60 con 0x7fb3b013dc30 2024-08-09T14:50:51.048 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.037+0000 7fb397fff640 10 client.4338.objecter ms_dispatch 0x7fb3b0000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:51.048 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.037+0000 7fb397fff640 3 client.4338.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:51.048 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.037+0000 7fb397fff640 3 client.4338.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:51.048 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.037+0000 7fb397fff640 20 client.4338.objecter dump_active .. 0 homeless 2024-08-09T14:50:51.048 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.037+0000 7fb397fff640 1 -- 172.21.15.155:0/4152612314 <== mon.0 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7fb3b00b0ab0 con 0x7fb3b013dc30 2024-08-09T14:50:51.048 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.037+0000 7fb397fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.048 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.037+0000 7fb397fff640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:51.049 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.040+0000 7fdac545a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdab80cac70 0x7fdab8160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:51.049 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.040+0000 7fdac545a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdab80cac70 0x7fdab8160a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:47984/0 (socket says 172.21.15.155:47984) 2024-08-09T14:50:51.049 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.040+0000 7fdac545a640 1 -- 172.21.15.155:0/3868564001 learned_addr learned my addr 172.21.15.155:0/3868564001 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:51.049 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.040+0000 7fdac545a640 10 monclient(hunting): get_auth_request con 0x7fdab80cac70 auth_method 0 2024-08-09T14:50:51.049 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.040+0000 7fdac545a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:51.049 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.040+0000 7fdac545a640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:51.049 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.040+0000 7fdac545a640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:51.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f1918952640 1 -- 172.21.15.155:0/1728482897 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910038070 con 0x7f19140b0160 2024-08-09T14:50:51.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f1918952640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:51.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f1918952640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:51.050 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f1918952640 10 monclient: dump: 2024-08-09T14:50:51.050 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:51.050 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:51.050 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f1918952640 10 monclient: _finish_auth 0 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f1918952640 10 monclient: _check_auth_tickets 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f1918952640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f1918952640 1 -- 172.21.15.155:0/1728482897 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1910031740 con 0x7f19140b0160 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f191bbb8640 5 monclient: authenticate success, global_id 4330 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f191bbb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f1918952640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:51.051 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f1918952640 1 -- 172.21.15.155:0/1728482897 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f191002f050 con 0x7f19140b0160 2024-08-09T14:50:51.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f1918952640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:51.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f1918952640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:51.052 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f1918952640 10 monclient: dump: 2024-08-09T14:50:51.052 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:51.052 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:51.052 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.052 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.052 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:51.052 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.052 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:51.052 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:51.052 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:51.052 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f191bbb8640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:51.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7f191bbb8640 4 set_mon_vals no callback set 2024-08-09T14:50:51.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7fdac545a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:51.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7fdac545a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:51.053 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.041+0000 7fdac545a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:51.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.042+0000 7f191bbb8640 10 monclient: shutdown 2024-08-09T14:50:51.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.042+0000 7f191bbb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:51.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.042+0000 7f191bbb8640 1 -- 172.21.15.155:0/1728482897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19140b0160 msgr2=0x7f19140b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.042+0000 7f191bbb8640 1 --2- 172.21.15.155:0/1728482897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19140b0160 0x7f19140b0540 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1910009870 tx=0x7f191002b7e0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.042+0000 7f191bbb8640 1 -- 172.21.15.155:0/1728482897 shutdown_connections 2024-08-09T14:50:51.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.042+0000 7f191bbb8640 1 --2- 172.21.15.155:0/1728482897 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19140b22e0 0x7f19141469d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.042+0000 7f191bbb8640 1 --2- 172.21.15.155:0/1728482897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19140b0160 0x7f19140b0540 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.042+0000 7f191bbb8640 1 -- 172.21.15.155:0/1728482897 >> 172.21.15.155:0/1728482897 conn(0x7f191406dbd0 msgr2=0x7f19140676f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:51.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.042+0000 7f191bbb8640 1 -- 172.21.15.155:0/1728482897 shutdown_connections 2024-08-09T14:50:51.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.042+0000 7f191bbb8640 1 -- 172.21.15.155:0/1728482897 wait complete. 2024-08-09T14:50:51.055 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.043+0000 7f191bbb8640 10 monclient: build_initial_monmap 2024-08-09T14:50:51.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.043+0000 7f191bbb8640 10 monclient: monmap: 2024-08-09T14:50:51.056 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:51.056 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:51.056 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:51.056 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:51.056 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:51.056 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.056 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:51.056 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:51.056 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:51.056 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.056 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.043+0000 7f191bbb8640 1 librados: starting msgr at 2024-08-09T14:50:51.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.043+0000 7f191bbb8640 1 librados: starting objecter 2024-08-09T14:50:51.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7f191bbb8640 1 Processor -- start 2024-08-09T14:50:51.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7f191bbb8640 1 -- start start 2024-08-09T14:50:51.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7f191bbb8640 1 librados: setting wanted keys 2024-08-09T14:50:51.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7f191bbb8640 1 librados: calling monclient init 2024-08-09T14:50:51.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7f191bbb8640 10 monclient: init 2024-08-09T14:50:51.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7fdac545a640 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2024-08-09T14:50:51.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7fdac545a640 1 -- 172.21.15.155:0/3868564001 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdab8039210 msgr2=0x7fdab802fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7fdac545a640 1 --2- 172.21.15.155:0/3868564001 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdab8039210 0x7fdab802fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7fdac545a640 1 -- 172.21.15.155:0/3868564001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdab80c46d0 msgr2=0x7fdab80c5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7fdac545a640 1 --2- 172.21.15.155:0/3868564001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdab80c46d0 0x7fdab80c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7fdac545a640 10 monclient: _finish_hunting 0 2024-08-09T14:50:51.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7fdac545a640 1 monclient: found mon.noname-b 2024-08-09T14:50:51.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7fdac545a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:51.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7fdac545a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:51.058 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7fdac545a640 1 -- 172.21.15.155:0/3868564001 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab81602a0 con 0x7fdab80cac70 2024-08-09T14:50:51.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.044+0000 7f191bbb8640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:51.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 10 monclient: _add_conns ranks=[1,2,0] 2024-08-09T14:50:51.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19140b0160 0x7f19140aa700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:51.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 10 monclient(hunting): picked mon.noname-b con 0x7f19140b0160 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:51.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19140b22e0 0x7f1914149060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:51.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 10 monclient(hunting): picked mon.noname-c con 0x7f19140b22e0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:51.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1914146f10 0x7f19141db7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:51.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 10 monclient(hunting): picked mon.noname-a con 0x7f1914146f10 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:51.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:51.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f191414d6d0 con 0x7f1914146f10 2024-08-09T14:50:51.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:51.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f191414d550 con 0x7f19140b0160 2024-08-09T14:50:51.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:51.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f191414d850 con 0x7f19140b22e0 2024-08-09T14:50:51.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:51.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.045+0000 7f191bbb8640 10 monclient(hunting): authenticate will time out at 1258.082271s 2024-08-09T14:50:51.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.046+0000 7f65be7fc640 1 -- 172.21.15.155:0/1311888196 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f65c0023e00 con 0x7f65c81444e0 2024-08-09T14:50:51.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.046+0000 7f65be7fc640 10 client.4304.objecter ms_dispatch 0x7f65c8000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:51.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.046+0000 7f65be7fc640 3 client.4304.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:51.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.046+0000 7f65be7fc640 3 client.4304.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:51.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.046+0000 7f65be7fc640 20 client.4304.objecter dump_active .. 0 homeless 2024-08-09T14:50:51.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.047+0000 7f65be7fc640 1 -- 172.21.15.155:0/1311888196 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f65c80b5dd0 con 0x7f65c81444e0 2024-08-09T14:50:51.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.047+0000 7f65be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.047+0000 7f65be7fc640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:51.062 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.048+0000 7f1919954640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19140b0160 0x7f19140aa700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:51.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.048+0000 7fdac545a640 1 --2- 172.21.15.155:0/3868564001 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdab80cac70 0x7fdab8160a60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdac00746f0 tx=0x7fdac006de70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:51.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.048+0000 7fdabd7fa640 10 client.?.objecter ms_handle_connect 0x7fdab80cac70 2024-08-09T14:50:51.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.048+0000 7fdabd7fa640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:51.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.049+0000 7f1919954640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19140b0160 0x7f19140aa700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:47990/0 (socket says 172.21.15.155:47990) 2024-08-09T14:50:51.063 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.049+0000 7f1919954640 1 -- 172.21.15.155:0/4261962391 learned_addr learned my addr 172.21.15.155:0/4261962391 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:51.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.049+0000 7f1919954640 10 monclient(hunting): get_auth_request con 0x7f19140b0160 auth_method 0 2024-08-09T14:50:51.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.049+0000 7f1919954640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:51.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.049+0000 7f1919954640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:51.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.049+0000 7f1919954640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:51.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.049+0000 7f6e9154a640 1 --2- 172.21.15.155:0/1044154739 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e8c0b22e0 0x7f6e8c06fe00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800e9b0 tx=0x7f6e8800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:51.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.049+0000 7f6e827fc640 10 client.?.objecter ms_handle_connect 0x7f6e8c0b22e0 2024-08-09T14:50:51.064 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.049+0000 7f6e827fc640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:51.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.050+0000 7f1919954640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:51.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.050+0000 7f1919954640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:51.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.050+0000 7f1919954640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:51.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.053+0000 7f1919954640 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2024-08-09T14:50:51.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.053+0000 7f1919954640 1 -- 172.21.15.155:0/4261962391 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19140b22e0 msgr2=0x7f1914149060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.053+0000 7f1919954640 1 --2- 172.21.15.155:0/4261962391 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19140b22e0 0x7f1914149060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.053+0000 7f1919954640 1 -- 172.21.15.155:0/4261962391 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1914146f10 msgr2=0x7f19141db7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.065 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.053+0000 7f1919954640 1 --2- 172.21.15.155:0/4261962391 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1914146f10 0x7f19141db7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.053+0000 7f1919954640 10 monclient: _finish_hunting 0 2024-08-09T14:50:51.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.053+0000 7f1919954640 1 monclient: found mon.noname-b 2024-08-09T14:50:51.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.053+0000 7f1919954640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:51.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.053+0000 7f1919954640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:51.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.053+0000 7f1919954640 1 -- 172.21.15.155:0/4261962391 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19140aba10 con 0x7f19140b0160 2024-08-09T14:50:51.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.057+0000 7f1919954640 1 --2- 172.21.15.155:0/4261962391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19140b0160 0x7f19140aa700 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1910009870 tx=0x7f19100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:51.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.057+0000 7f190a7fc640 10 client.?.objecter ms_handle_connect 0x7f19140b0160 2024-08-09T14:50:51.066 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.057+0000 7f190a7fc640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:51.067 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.057+0000 7fdabd7fa640 1 -- 172.21.15.155:0/3868564001 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac006aaa0 con 0x7fdab80cac70 2024-08-09T14:50:51.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.057+0000 7fdabd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:51.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.057+0000 7fdabd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:51.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.057+0000 7fdabd7fa640 10 monclient: dump: 2024-08-09T14:50:51.068 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:51.068 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:51.068 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.068 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.068 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:51.068 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.068 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:51.068 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:51.068 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:51.068 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.068 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.057+0000 7fdabd7fa640 10 monclient: _finish_auth 0 2024-08-09T14:50:51.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.057+0000 7fdabd7fa640 10 monclient: _check_auth_tickets 2024-08-09T14:50:51.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.058+0000 7fdabd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:51.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.058+0000 7fdabd7fa640 1 -- 172.21.15.155:0/3868564001 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdac0090070 con 0x7fdab80cac70 2024-08-09T14:50:51.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.058+0000 7fdac5c5b640 5 monclient: authenticate success, global_id 4333 2024-08-09T14:50:51.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.058+0000 7fdac5c5b640 10 monclient: _renew_subs 2024-08-09T14:50:51.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.058+0000 7fdac5c5b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:51.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.058+0000 7fdac5c5b640 1 -- 172.21.15.155:0/3868564001 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab802f2f0 con 0x7fdab80cac70 2024-08-09T14:50:51.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.058+0000 7fdabd7fa640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:51.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.058+0000 7fdac5c5b640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:51.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.058+0000 7fdabeffd640 4 set_mon_vals no callback set 2024-08-09T14:50:51.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.059+0000 7fdabd7fa640 1 -- 172.21.15.155:0/3868564001 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac0068960 con 0x7fdab80cac70 2024-08-09T14:50:51.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.059+0000 7fdabd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:51.069 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.059+0000 7fdabd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:51.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.059+0000 7fdabd7fa640 10 monclient: dump: 2024-08-09T14:50:51.070 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:51.070 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:51.070 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.070 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.070 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:51.070 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.070 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:51.070 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:51.070 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:51.070 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.070 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.059+0000 7fdac5c5b640 10 monclient: _renew_subs 2024-08-09T14:50:51.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.059+0000 7fdac5c5b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:51.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.059+0000 7fdac5c5b640 1 -- 172.21.15.155:0/3868564001 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab80c8270 con 0x7fdab80cac70 2024-08-09T14:50:51.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.059+0000 7fdac5c5b640 1 librados: init done 2024-08-09T14:50:51.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.060+0000 7fdaaaffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.060+0000 7fdaaaffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.060+0000 7fdaaaffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:51.071 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.060+0000 7fdaaaffd640 1 -- 172.21.15.155:0/3868564001 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda98005080 con 0x7fdab80cac70 2024-08-09T14:50:51.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f190a7fc640 1 -- 172.21.15.155:0/4261962391 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910043070 con 0x7f19140b0160 2024-08-09T14:50:51.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f190a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:51.072 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f190a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:51.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f190a7fc640 10 monclient: dump: 2024-08-09T14:50:51.073 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:51.073 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:51.073 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.073 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.073 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:51.073 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.073 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:51.073 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:51.073 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:51.073 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.073 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f190a7fc640 10 monclient: _finish_auth 0 2024-08-09T14:50:51.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f190a7fc640 10 monclient: _check_auth_tickets 2024-08-09T14:50:51.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f190a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:51.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f190a7fc640 1 -- 172.21.15.155:0/4261962391 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1910033a70 con 0x7f19140b0160 2024-08-09T14:50:51.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f191bbb8640 5 monclient: authenticate success, global_id 4336 2024-08-09T14:50:51.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f191bbb8640 10 monclient: _renew_subs 2024-08-09T14:50:51.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f191bbb8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:51.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f191bbb8640 1 -- 172.21.15.155:0/4261962391 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19140abcf0 con 0x7f19140b0160 2024-08-09T14:50:51.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f190a7fc640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:51.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f190a7fc640 1 -- 172.21.15.155:0/4261962391 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1910038040 con 0x7f19140b0160 2024-08-09T14:50:51.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f190a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:51.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f190a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:51.074 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f190a7fc640 10 monclient: dump: 2024-08-09T14:50:51.075 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:51.075 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:51.075 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.075 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.075 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:51.075 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.075 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:51.075 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:51.075 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:51.075 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.068+0000 7f190bfff640 4 set_mon_vals no callback set 2024-08-09T14:50:51.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.069+0000 7f191bbb8640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:51.075 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.069+0000 7f191bbb8640 10 monclient: _renew_subs 2024-08-09T14:50:51.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.069+0000 7f191bbb8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:51.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.069+0000 7f191bbb8640 1 -- 172.21.15.155:0/4261962391 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f191414c430 con 0x7f19140b0160 2024-08-09T14:50:51.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.069+0000 7f191bbb8640 1 librados: init done 2024-08-09T14:50:51.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.070+0000 7f191bbb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.070+0000 7f191bbb8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.070+0000 7f191bbb8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:51.076 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.070+0000 7f191bbb8640 1 -- 172.21.15.155:0/4261962391 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19140b0540 con 0x7f19140b0160 2024-08-09T14:50:51.081 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.080+0000 7fdabd7fa640 1 -- 172.21.15.155:0/3868564001 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fdac0068b20 con 0x7fdab80cac70 2024-08-09T14:50:51.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.081+0000 7fdabd7fa640 1 --2- 172.21.15.155:0/3868564001 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fdaa406d840 0x7fdaa406fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:51.082 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.081+0000 7fdac4c59640 1 --2- 172.21.15.155:0/3868564001 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fdaa406d840 0x7fdaa406fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:51.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.081+0000 7fdac4c59640 10 monclient: get_auth_request con 0x7fdaa406d840 auth_method 0 2024-08-09T14:50:51.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.082+0000 7fdabd7fa640 10 client.4333.objecter ms_handle_connect 0x7fdaa406d840 2024-08-09T14:50:51.083 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.082+0000 7fdac4c59640 1 --2- 172.21.15.155:0/3868564001 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fdaa406d840 0x7fdaa406fd00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdab4006a90 tx=0x7fdab4006a20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:51.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.240+0000 7f1b3937e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-08-09T14:50:51.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.240+0000 7f1b3937e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-08-09T14:50:51.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.240+0000 7f1b3937e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:51.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.240+0000 7f1b3937e640 1 -- 172.21.15.155:0/3609027866 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f1b34070c60 con 0x7f1b340b2d80 2024-08-09T14:50:51.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e827fc640 1 -- 172.21.15.155:0/1044154739 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88017070 con 0x7f6e8c0b22e0 2024-08-09T14:50:51.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:51.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:51.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e827fc640 10 monclient: dump: 2024-08-09T14:50:51.306 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:51.306 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:51.306 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.306 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.307 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:51.307 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.307 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:51.307 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:51.307 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:51.307 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e827fc640 10 monclient: _finish_auth 0 2024-08-09T14:50:51.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e827fc640 10 monclient: _check_auth_tickets 2024-08-09T14:50:51.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:51.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e93faf640 5 monclient: authenticate success, global_id 4310 2024-08-09T14:50:51.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e93faf640 10 monclient: _renew_subs 2024-08-09T14:50:51.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e93faf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:51.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e93faf640 1 -- 172.21.15.155:0/1044154739 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e8c071930 con 0x7f6e8c0b22e0 2024-08-09T14:50:51.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e93faf640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:51.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e93faf640 10 monclient: _renew_subs 2024-08-09T14:50:51.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e93faf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:51.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e93faf640 1 -- 172.21.15.155:0/1044154739 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e8c072c90 con 0x7f6e8c0b22e0 2024-08-09T14:50:51.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.304+0000 7f6e93faf640 1 librados: init done 2024-08-09T14:50:51.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.305+0000 7f6e93faf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.305+0000 7f6e93faf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.305+0000 7f6e93faf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:51.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.305+0000 7f6e93faf640 1 -- 172.21.15.155:0/1044154739 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e8c1469d0 con 0x7f6e8c0b22e0 2024-08-09T14:50:51.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.308+0000 7f6e827fc640 1 -- 172.21.15.155:0/1044154739 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e8800c9a0 con 0x7f6e8c0b22e0 2024-08-09T14:50:51.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.308+0000 7f6e827fc640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:51.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.308+0000 7f6e827fc640 1 -- 172.21.15.155:0/1044154739 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e88021060 con 0x7f6e8c0b22e0 2024-08-09T14:50:51.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.308+0000 7f6e827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:51.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.308+0000 7f6e827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:51.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.308+0000 7f6e827fc640 10 monclient: dump: 2024-08-09T14:50:51.310 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:51.310 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:51.310 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.310 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:51.311 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:51.311 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:51.311 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:51.311 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:51.311 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:51.311 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:51.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.308+0000 7f6e83fff640 4 set_mon_vals no callback set 2024-08-09T14:50:51.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.325+0000 7f4405ffb640 1 -- 172.21.15.155:0/1336843718 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f440801a070 con 0x7f44101470c0 2024-08-09T14:50:51.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.325+0000 7f4405ffb640 1 --2- 172.21.15.155:0/1336843718 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f43f406d7c0 0x7f43f406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:51.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.325+0000 7f441551f640 1 --2- 172.21.15.155:0/1336843718 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f43f406d7c0 0x7f43f406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:51.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.325+0000 7f441551f640 10 monclient: get_auth_request con 0x7f43f406d7c0 auth_method 0 2024-08-09T14:50:51.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.326+0000 7f4405ffb640 10 client.4307.objecter ms_handle_connect 0x7f43f406d7c0 2024-08-09T14:50:51.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.326+0000 7f441551f640 1 --2- 172.21.15.155:0/1336843718 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f43f406d7c0 0x7f43f406fc80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f440c009560 tx=0x7f440c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:51.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.339+0000 7f4405ffb640 1 -- 172.21.15.155:0/1336843718 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f4408057240 con 0x7f44101470c0 2024-08-09T14:50:51.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.339+0000 7f4405ffb640 10 client.4307.objecter ms_dispatch 0x7f4410000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:51.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.339+0000 7f4405ffb640 3 client.4307.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:51.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.339+0000 7f4405ffb640 3 client.4307.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:51.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.339+0000 7f4405ffb640 20 client.4307.objecter dump_active .. 0 homeless 2024-08-09T14:50:51.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.339+0000 7f4405ffb640 1 -- 172.21.15.155:0/1336843718 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f4408021020 con 0x7f44101470c0 2024-08-09T14:50:51.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.339+0000 7f4405ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.339+0000 7f4405ffb640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:51.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.361+0000 7f65cfc33640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-09T14:50:51.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.361+0000 7f65cfc33640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-09T14:50:51.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.361+0000 7f65cfc33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:51.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.361+0000 7f65cfc33640 1 -- 172.21.15.155:0/1311888196 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f65c81323a0 con 0x7f65c81444e0 2024-08-09T14:50:51.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.364+0000 7fb3b587b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-08-09T14:50:51.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.364+0000 7fb3b587b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-08-09T14:50:51.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.364+0000 7fb3b587b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:51.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.364+0000 7fb3b587b640 1 -- 172.21.15.155:0/4152612314 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb3b0145c50 con 0x7fb3b013dc30 2024-08-09T14:50:51.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.375+0000 7f99417fa640 1 -- 172.21.15.155:0/2389289502 <== mon.0 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f993c014960 con 0x7f99441465f0 2024-08-09T14:50:51.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.376+0000 7f99417fa640 1 --2- 172.21.15.155:0/2389289502 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f992806d7c0 0x7f992806fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:51.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.376+0000 7f9948f2a640 1 --2- 172.21.15.155:0/2389289502 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f992806d7c0 0x7f992806fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:51.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.376+0000 7f9948f2a640 10 monclient: get_auth_request con 0x7f992806d7c0 auth_method 0 2024-08-09T14:50:51.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.376+0000 7f99417fa640 10 client.4341.objecter ms_handle_connect 0x7f992806d7c0 2024-08-09T14:50:51.378 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.376+0000 7f9948f2a640 1 --2- 172.21.15.155:0/2389289502 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f992806d7c0 0x7f992806fc80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f99380017f0 tx=0x7f9938009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:51.394 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.392+0000 7f99417fa640 1 -- 172.21.15.155:0/2389289502 <== mon.0 v2:172.21.15.155:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f993c058830 con 0x7f99441465f0 2024-08-09T14:50:51.394 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.392+0000 7f99417fa640 10 client.4341.objecter ms_dispatch 0x7f9944000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:51.394 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.392+0000 7f99417fa640 3 client.4341.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:51.394 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.392+0000 7f99417fa640 3 client.4341.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:51.394 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.392+0000 7f99417fa640 20 client.4341.objecter dump_active .. 0 homeless 2024-08-09T14:50:51.394 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.392+0000 7f99417fa640 1 -- 172.21.15.155:0/2389289502 <== mon.0 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f99440b0b30 con 0x7f99441465f0 2024-08-09T14:50:51.394 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.392+0000 7f99417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.394 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.392+0000 7f99417fa640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:51.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.432+0000 7fb397fff640 1 -- 172.21.15.155:0/4152612314 <== mon.0 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb3a807c4c0 con 0x7fb3b013dc30 2024-08-09T14:50:51.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.432+0000 7fb397fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-08-09T14:50:51.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.432+0000 7fb397fff640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:51.434 INFO:teuthology.orchestra.run.smithi155.stdout:60129542150 2024-08-09T14:50:51.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.434+0000 7fb3b587b640 10 librados: watch_flush enter 2024-08-09T14:50:51.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.434+0000 7fb3b587b640 10 librados: watch_flush exit 2024-08-09T14:50:51.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.434+0000 7fb3b587b640 20 client.4338.objecter shutdown clearing up homeless session... 2024-08-09T14:50:51.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.434+0000 7fb3b587b640 10 client.4338.objecter successfully canceled tick 2024-08-09T14:50:51.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.434+0000 7fb3b587b640 1 -- 172.21.15.155:0/4152612314 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fb39c06d810 msgr2=0x7fb39c06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.434+0000 7fb3b587b640 1 --2- 172.21.15.155:0/4152612314 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fb39c06d810 0x7fb39c06fcd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0006a90 tx=0x7fb3a0006a20 comp rx=0 tx=0).stop 2024-08-09T14:50:51.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.435+0000 7fb3b587b640 10 monclient: shutdown 2024-08-09T14:50:51.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.435+0000 7fb3b587b640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:51.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.435+0000 7fb3b587b640 1 -- 172.21.15.155:0/4152612314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b013dc30 msgr2=0x7fb3b0145710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.435+0000 7fb3b587b640 1 --2- 172.21.15.155:0/4152612314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b013dc30 0x7fb3b0145710 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009f70 tx=0x7fb3a8004a40 comp rx=0 tx=0).stop 2024-08-09T14:50:51.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.435+0000 7fb3b587b640 1 -- 172.21.15.155:0/4152612314 shutdown_connections 2024-08-09T14:50:51.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.435+0000 7fb3b587b640 1 --2- 172.21.15.155:0/4152612314 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fb39c06d810 0x7fb39c06fcd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.435+0000 7fb3b587b640 1 --2- 172.21.15.155:0/4152612314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb3b0144240 0x7fb3b01410f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.435+0000 7fb3b587b640 1 --2- 172.21.15.155:0/4152612314 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fb3b013dc30 0x7fb3b0145710 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.435+0000 7fb3b587b640 1 --2- 172.21.15.155:0/4152612314 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fb3b00b1150 0x7fb3b00b5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.435+0000 7fb3b587b640 1 -- 172.21.15.155:0/4152612314 >> 172.21.15.155:0/4152612314 conn(0x7fb3b00680f0 msgr2=0x7fb3b013a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:51.437 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.435+0000 7fb3b587b640 1 -- 172.21.15.155:0/4152612314 shutdown_connections 2024-08-09T14:50:51.438 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.435+0000 7fb3b587b640 1 -- 172.21.15.155:0/4152612314 wait complete. 2024-08-09T14:50:51.438 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.435+0000 7fb3b587b640 1 librados: shutdown 2024-08-09T14:50:51.448 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.6 2024-08-09T14:50:51.551 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.549+0000 7f4417783640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-09T14:50:51.551 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.549+0000 7f4417783640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-09T14:50:51.551 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.549+0000 7f4417783640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:51.551 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.549+0000 7f4417783640 1 -- 172.21.15.155:0/1336843718 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f4410070030 con 0x7f44101470c0 2024-08-09T14:50:51.559 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.558+0000 7fdabd7fa640 1 -- 172.21.15.155:0/3868564001 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7fdac00d2690 con 0x7fdab80cac70 2024-08-09T14:50:51.560 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.558+0000 7fdabd7fa640 10 client.4333.objecter ms_dispatch 0x7fdac0000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:51.560 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.558+0000 7fdabd7fa640 3 client.4333.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:51.560 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.558+0000 7fdabd7fa640 3 client.4333.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:51.560 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.558+0000 7fdabd7fa640 20 client.4333.objecter dump_active .. 0 homeless 2024-08-09T14:50:51.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.559+0000 7f190a7fc640 1 -- 172.21.15.155:0/4261962391 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1910046070 con 0x7f19140b0160 2024-08-09T14:50:51.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.559+0000 7f190a7fc640 1 --2- 172.21.15.155:0/4261962391 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f18f806d840 0x7f18f806fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:51.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.559+0000 7f1919153640 1 --2- 172.21.15.155:0/4261962391 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f18f806d840 0x7f18f806fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:51.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.559+0000 7f1919153640 10 monclient: get_auth_request con 0x7f18f806d840 auth_method 0 2024-08-09T14:50:51.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.560+0000 7f190a7fc640 10 client.4336.objecter ms_handle_connect 0x7f18f806d840 2024-08-09T14:50:51.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.560+0000 7f1919153640 1 --2- 172.21.15.155:0/4261962391 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f18f806d840 0x7f18f806fd00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f191414c220 tx=0x7f1904006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:51.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.564+0000 7f190a7fc640 1 -- 172.21.15.155:0/4261962391 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f19100791e0 con 0x7f19140b0160 2024-08-09T14:50:51.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.564+0000 7f190a7fc640 10 client.4336.objecter ms_dispatch 0x7f1914000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:51.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.564+0000 7f190a7fc640 3 client.4336.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:51.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.564+0000 7f190a7fc640 3 client.4336.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:51.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.564+0000 7f190a7fc640 20 client.4336.objecter dump_active .. 0 homeless 2024-08-09T14:50:51.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.574+0000 7f190a7fc640 1 -- 172.21.15.155:0/4261962391 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f191007b840 con 0x7f19140b0160 2024-08-09T14:50:51.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.574+0000 7f190a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.574+0000 7f190a7fc640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:51.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.585+0000 7fdabd7fa640 1 -- 172.21.15.155:0/3868564001 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7fdac00a4080 con 0x7fdab80cac70 2024-08-09T14:50:51.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.585+0000 7fdabd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.586 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.585+0000 7fdabd7fa640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:51.588 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.586+0000 7f9949f2c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-09T14:50:51.588 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.586+0000 7f9949f2c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-09T14:50:51.588 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.586+0000 7f9949f2c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:51.588 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.586+0000 7f9949f2c640 1 -- 172.21.15.155:0/2389289502 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f9944072e20 con 0x7f99441465f0 2024-08-09T14:50:51.591 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.590+0000 7f1b1b7fe640 1 -- 172.21.15.155:0/3609027866 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1b34070c60 con 0x7f1b340b2d80 2024-08-09T14:50:51.591 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.590+0000 7f1b1b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-08-09T14:50:51.591 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.590+0000 7f1b1b7fe640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:51.591 INFO:teuthology.orchestra.run.smithi155.stdout:81604378633 2024-08-09T14:50:51.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 10 librados: watch_flush enter 2024-08-09T14:50:51.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 10 librados: watch_flush exit 2024-08-09T14:50:51.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 20 client.4318.objecter shutdown clearing up homeless session... 2024-08-09T14:50:51.594 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 10 client.4318.objecter successfully canceled tick 2024-08-09T14:50:51.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 1 -- 172.21.15.155:0/3609027866 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f1b2006d7c0 msgr2=0x7f1b2006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 1 --2- 172.21.15.155:0/3609027866 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f1b2006d7c0 0x7f1b2006fc80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0012e0 tx=0x7f1b2c009040 comp rx=0 tx=0).stop 2024-08-09T14:50:51.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 10 monclient: shutdown 2024-08-09T14:50:51.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:51.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 1 -- 172.21.15.155:0/3609027866 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b340b2d80 msgr2=0x7f1b34145200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 1 --2- 172.21.15.155:0/3609027866 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b340b2d80 0x7f1b34145200 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1b28009890 tx=0x7f1b280046c0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 1 -- 172.21.15.155:0/3609027866 shutdown_connections 2024-08-09T14:50:51.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 1 --2- 172.21.15.155:0/3609027866 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f1b2006d7c0 0x7f1b2006fc80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 1 --2- 172.21.15.155:0/3609027866 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1b340b4df0 0x7f1b34145740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 1 --2- 172.21.15.155:0/3609027866 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1b340b2d80 0x7f1b34145200 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 1 --2- 172.21.15.155:0/3609027866 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f1b3406d8d0 0x7f1b340701f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.596 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 1 -- 172.21.15.155:0/3609027866 >> 172.21.15.155:0/3609027866 conn(0x7f1b340b8d70 msgr2=0x7f1b340b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:51.597 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f99417fa640 1 -- 172.21.15.155:0/2389289502 <== mon.0 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f993c05ae90 con 0x7f99441465f0 2024-08-09T14:50:51.597 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f99417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-09T14:50:51.597 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f99417fa640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:51.598 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 1 -- 172.21.15.155:0/3609027866 shutdown_connections 2024-08-09T14:50:51.598 INFO:teuthology.orchestra.run.smithi155.stdout:60129542150 2024-08-09T14:50:51.598 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.592+0000 7f1b3937e640 1 -- 172.21.15.155:0/3609027866 wait complete. 2024-08-09T14:50:51.599 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.593+0000 7f1b3937e640 1 librados: shutdown 2024-08-09T14:50:51.601 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 10 librados: watch_flush enter 2024-08-09T14:50:51.601 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 10 librados: watch_flush exit 2024-08-09T14:50:51.601 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 20 client.4341.objecter shutdown clearing up homeless session... 2024-08-09T14:50:51.601 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 10 client.4341.objecter successfully canceled tick 2024-08-09T14:50:51.601 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 1 -- 172.21.15.155:0/2389289502 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f992806d7c0 msgr2=0x7f992806fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.601 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 1 --2- 172.21.15.155:0/2389289502 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f992806d7c0 0x7f992806fc80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f99380017f0 tx=0x7f9938009040 comp rx=0 tx=0).stop 2024-08-09T14:50:51.602 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 10 monclient: shutdown 2024-08-09T14:50:51.602 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:51.602 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 1 -- 172.21.15.155:0/2389289502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99441465f0 msgr2=0x7f994413b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.602 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 1 --2- 172.21.15.155:0/2389289502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99441465f0 0x7f994413b530 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f993c007e60 tx=0x7f993c00e4b0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.602 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 1 -- 172.21.15.155:0/2389289502 shutdown_connections 2024-08-09T14:50:51.602 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 1 --2- 172.21.15.155:0/2389289502 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f992806d7c0 0x7f992806fc80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.602 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 1 --2- 172.21.15.155:0/2389289502 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f99441465f0 0x7f994413b530 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.602 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 1 --2- 172.21.15.155:0/2389289502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9944140510 0x7f994413aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.602 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 1 --2- 172.21.15.155:0/2389289502 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f994406d320 0x7f994413f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.595+0000 7f9949f2c640 1 -- 172.21.15.155:0/2389289502 >> 172.21.15.155:0/2389289502 conn(0x7f9944068300 msgr2=0x7f994413fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:51.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.596+0000 7f9949f2c640 1 -- 172.21.15.155:0/2389289502 shutdown_connections 2024-08-09T14:50:51.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.596+0000 7f9949f2c640 1 -- 172.21.15.155:0/2389289502 wait complete. 2024-08-09T14:50:51.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.596+0000 7f9949f2c640 1 librados: shutdown 2024-08-09T14:50:51.609 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378635 got 81604378633 for osd.7 2024-08-09T14:50:51.615 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.4 2024-08-09T14:50:51.624 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.623+0000 7f6e827fc640 1 -- 172.21.15.155:0/1044154739 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f6e8801a070 con 0x7f6e8c0b22e0 2024-08-09T14:50:51.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.623+0000 7f6e827fc640 1 --2- 172.21.15.155:0/1044154739 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f6e7006d7c0 0x7f6e7006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:51.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.624+0000 7f6e91d4b640 1 --2- 172.21.15.155:0/1044154739 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f6e7006d7c0 0x7f6e7006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:51.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.624+0000 7f6e91d4b640 10 monclient: get_auth_request con 0x7f6e7006d7c0 auth_method 0 2024-08-09T14:50:51.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.624+0000 7f6e827fc640 10 client.4310.objecter ms_handle_connect 0x7f6e7006d7c0 2024-08-09T14:50:51.625 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.624+0000 7f6e91d4b640 1 --2- 172.21.15.155:0/1044154739 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f6e7006d7c0 0x7f6e7006fc80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c035000 tx=0x7f6e7c030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:51.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.644+0000 7f6e827fc640 1 -- 172.21.15.155:0/1044154739 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f6e880579d0 con 0x7f6e8c0b22e0 2024-08-09T14:50:51.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.644+0000 7f6e827fc640 10 client.4310.objecter ms_dispatch 0x7f6e8c000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:51.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.644+0000 7f6e827fc640 3 client.4310.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:51.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.644+0000 7f6e827fc640 3 client.4310.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:51.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.644+0000 7f6e827fc640 20 client.4310.objecter dump_active .. 0 homeless 2024-08-09T14:50:51.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.644+0000 7f6e827fc640 1 -- 172.21.15.155:0/1044154739 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f6e8805a030 con 0x7f6e8c0b22e0 2024-08-09T14:50:51.646 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.644+0000 7f6e827fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:51.647 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.644+0000 7f6e827fc640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:51.658 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.657+0000 7f65be7fc640 1 -- 172.21.15.155:0/1311888196 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f65c0020020 con 0x7f65c81444e0 2024-08-09T14:50:51.658 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.657+0000 7f65be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-09T14:50:51.659 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.657+0000 7f65be7fc640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:51.659 INFO:teuthology.orchestra.run.smithi155.stdout:64424509445 2024-08-09T14:50:51.659 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.657+0000 7f4405ffb640 1 -- 172.21.15.155:0/1336843718 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f44080591a0 con 0x7f44101470c0 2024-08-09T14:50:51.660 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.657+0000 7f4405ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-09T14:50:51.660 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.657+0000 7f4405ffb640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:51.660 INFO:teuthology.orchestra.run.smithi155.stdout:85899345928 2024-08-09T14:50:51.662 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 10 librados: watch_flush enter 2024-08-09T14:50:51.662 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 10 librados: watch_flush exit 2024-08-09T14:50:51.662 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 20 client.4304.objecter shutdown clearing up homeless session... 2024-08-09T14:50:51.662 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 10 client.4304.objecter successfully canceled tick 2024-08-09T14:50:51.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 1 -- 172.21.15.155:0/1311888196 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f65ac06d7f0 msgr2=0x7f65ac06fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 1 --2- 172.21.15.155:0/1311888196 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f65ac06d7f0 0x7f65ac06fcb0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f65c4005880 tx=0x7f65c4009040 comp rx=0 tx=0).stop 2024-08-09T14:50:51.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 10 monclient: shutdown 2024-08-09T14:50:51.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:51.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 1 -- 172.21.15.155:0/1311888196 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65c81444e0 msgr2=0x7f65c81326b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 1 --2- 172.21.15.155:0/1311888196 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65c81444e0 0x7f65c81326b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f65c000c730 tx=0x7f65c000cb30 comp rx=0 tx=0).stop 2024-08-09T14:50:51.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 1 -- 172.21.15.155:0/1311888196 shutdown_connections 2024-08-09T14:50:51.663 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 1 --2- 172.21.15.155:0/1311888196 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f65ac06d7f0 0x7f65ac06fcb0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.664 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 1 --2- 172.21.15.155:0/1311888196 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f65c81444e0 0x7f65c81326b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.664 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 1 --2- 172.21.15.155:0/1311888196 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f65c80b4ef0 0x7f65c812f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.664 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 1 --2- 172.21.15.155:0/1311888196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f65c80b0160 0x7f65c812f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.664 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 1 -- 172.21.15.155:0/1311888196 >> 172.21.15.155:0/1311888196 conn(0x7f65c806dbd0 msgr2=0x7f65c80b7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:51.664 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 1 -- 172.21.15.155:0/1311888196 shutdown_connections 2024-08-09T14:50:51.664 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 1 -- 172.21.15.155:0/1311888196 wait complete. 2024-08-09T14:50:51.664 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.659+0000 7f65cfc33640 1 librados: shutdown 2024-08-09T14:50:51.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 10 librados: watch_flush enter 2024-08-09T14:50:51.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 10 librados: watch_flush exit 2024-08-09T14:50:51.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 20 client.4307.objecter shutdown clearing up homeless session... 2024-08-09T14:50:51.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 10 client.4307.objecter successfully canceled tick 2024-08-09T14:50:51.665 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 1 -- 172.21.15.155:0/1336843718 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f43f406d7c0 msgr2=0x7f43f406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 1 --2- 172.21.15.155:0/1336843718 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f43f406d7c0 0x7f43f406fc80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f440c009560 tx=0x7f440c008030 comp rx=0 tx=0).stop 2024-08-09T14:50:51.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 10 monclient: shutdown 2024-08-09T14:50:51.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:51.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 1 -- 172.21.15.155:0/1336843718 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44101470c0 msgr2=0x7f44101dd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 1 --2- 172.21.15.155:0/1336843718 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44101470c0 0x7f44101dd0b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f440800caa0 tx=0x7f440800cf60 comp rx=0 tx=0).stop 2024-08-09T14:50:51.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 1 -- 172.21.15.155:0/1336843718 shutdown_connections 2024-08-09T14:50:51.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 1 --2- 172.21.15.155:0/1336843718 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f43f406d7c0 0x7f43f406fc80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 1 --2- 172.21.15.155:0/1336843718 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f44101470c0 0x7f44101dd0b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 1 --2- 172.21.15.155:0/1336843718 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4410072fa0 0x7f44101429d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 1 --2- 172.21.15.155:0/1336843718 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f441006bb80 0x7f44100b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.660+0000 7f4417783640 1 -- 172.21.15.155:0/1336843718 >> 172.21.15.155:0/1336843718 conn(0x7f44100b5af0 msgr2=0x7f4410146200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:51.666 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.661+0000 7f4417783640 1 -- 172.21.15.155:0/1336843718 shutdown_connections 2024-08-09T14:50:51.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.662+0000 7f4417783640 1 -- 172.21.15.155:0/1336843718 wait complete. 2024-08-09T14:50:51.667 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.662+0000 7f4417783640 1 librados: shutdown 2024-08-09T14:50:51.677 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.1 2024-08-09T14:50:51.680 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345930 got 85899345928 for osd.2 2024-08-09T14:50:51.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.789+0000 7f191bbb8640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-09T14:50:51.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.789+0000 7f191bbb8640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-09T14:50:51.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.789+0000 7f191bbb8640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:51.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.789+0000 7f191bbb8640 1 -- 172.21.15.155:0/4261962391 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f19140aac40 con 0x7f19140b0160 2024-08-09T14:50:51.807 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.805+0000 7fdaaaffd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-09T14:50:51.808 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.805+0000 7fdaaaffd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-09T14:50:51.808 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.805+0000 7fdaaaffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:51.808 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.805+0000 7fdaaaffd640 1 -- 172.21.15.155:0/3868564001 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fda98005640 con 0x7fdab80cac70 2024-08-09T14:50:51.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.827+0000 7f190a7fc640 1 -- 172.21.15.155:0/4261962391 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f191007b140 con 0x7f19140b0160 2024-08-09T14:50:51.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.828+0000 7f190a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-09T14:50:51.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.828+0000 7f190a7fc640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:51.829 INFO:teuthology.orchestra.run.smithi155.stdout:55834574854 2024-08-09T14:50:51.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.829+0000 7fdabd7fa640 1 -- 172.21.15.155:0/3868564001 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdac00d4cf0 con 0x7fdab80cac70 2024-08-09T14:50:51.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.829+0000 7fdabd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-09T14:50:51.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.829+0000 7fdabd7fa640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:51.830 INFO:teuthology.orchestra.run.smithi155.stdout:60129542149 2024-08-09T14:50:51.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 10 librados: watch_flush enter 2024-08-09T14:50:51.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 10 librados: watch_flush exit 2024-08-09T14:50:51.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 20 client.4336.objecter shutdown clearing up homeless session... 2024-08-09T14:50:51.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 10 client.4336.objecter successfully canceled tick 2024-08-09T14:50:51.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 1 -- 172.21.15.155:0/4261962391 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f18f806d840 msgr2=0x7f18f806fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 1 --2- 172.21.15.155:0/4261962391 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f18f806d840 0x7f18f806fd00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f191414c220 tx=0x7f1904006c90 comp rx=0 tx=0).stop 2024-08-09T14:50:51.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 10 monclient: shutdown 2024-08-09T14:50:51.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:51.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 1 -- 172.21.15.155:0/4261962391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19140b0160 msgr2=0x7f19140aa700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 1 --2- 172.21.15.155:0/4261962391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19140b0160 0x7f19140aa700 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1910009870 tx=0x7f19100046c0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 1 -- 172.21.15.155:0/4261962391 shutdown_connections 2024-08-09T14:50:51.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 1 --2- 172.21.15.155:0/4261962391 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f18f806d840 0x7f18f806fd00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 1 --2- 172.21.15.155:0/4261962391 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f1914146f10 0x7f19141db7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 1 --2- 172.21.15.155:0/4261962391 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f19140b22e0 0x7f1914149060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 1 --2- 172.21.15.155:0/4261962391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19140b0160 0x7f19140aa700 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 1 -- 172.21.15.155:0/4261962391 >> 172.21.15.155:0/4261962391 conn(0x7f191406dbd0 msgr2=0x7f1914140bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:51.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 1 -- 172.21.15.155:0/4261962391 shutdown_connections 2024-08-09T14:50:51.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 1 -- 172.21.15.155:0/4261962391 wait complete. 2024-08-09T14:50:51.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.830+0000 7f191bbb8640 1 librados: shutdown 2024-08-09T14:50:51.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 10 librados: watch_flush enter 2024-08-09T14:50:51.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 10 librados: watch_flush exit 2024-08-09T14:50:51.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 20 client.4333.objecter shutdown clearing up homeless session... 2024-08-09T14:50:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 10 client.4333.objecter successfully canceled tick 2024-08-09T14:50:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 1 -- 172.21.15.155:0/3868564001 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fdaa406d840 msgr2=0x7fdaa406fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 1 --2- 172.21.15.155:0/3868564001 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fdaa406d840 0x7fdaa406fd00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fdab4006a90 tx=0x7fdab4006a20 comp rx=0 tx=0).stop 2024-08-09T14:50:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 10 monclient: shutdown 2024-08-09T14:50:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 1 -- 172.21.15.155:0/3868564001 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdab80cac70 msgr2=0x7fdab8160a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 1 --2- 172.21.15.155:0/3868564001 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdab80cac70 0x7fdab8160a60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdac00746f0 tx=0x7fdac006de70 comp rx=0 tx=0).stop 2024-08-09T14:50:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 1 -- 172.21.15.155:0/3868564001 shutdown_connections 2024-08-09T14:50:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 1 --2- 172.21.15.155:0/3868564001 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fdaa406d840 0x7fdaa406fd00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 1 --2- 172.21.15.155:0/3868564001 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdab80cac70 0x7fdab8160a60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 1 --2- 172.21.15.155:0/3868564001 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fdab80c46d0 0x7fdab80c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 1 --2- 172.21.15.155:0/3868564001 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fdab8039210 0x7fdab802fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 1 -- 172.21.15.155:0/3868564001 >> 172.21.15.155:0/3868564001 conn(0x7fdab8006c00 msgr2=0x7fdab8034720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:51.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 1 -- 172.21.15.155:0/3868564001 shutdown_connections 2024-08-09T14:50:51.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 1 -- 172.21.15.155:0/3868564001 wait complete. 2024-08-09T14:50:51.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.831+0000 7fdaaaffd640 1 librados: shutdown 2024-08-09T14:50:51.844 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574854 for osd.3 2024-08-09T14:50:51.846 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.5 2024-08-09T14:50:51.855 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.854+0000 7f6e93faf640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-09T14:50:51.855 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.854+0000 7f6e93faf640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-09T14:50:51.855 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.854+0000 7f6e93faf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:51.855 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.854+0000 7f6e93faf640 1 -- 172.21.15.155:0/1044154739 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f6e8c068960 con 0x7f6e8c0b22e0 2024-08-09T14:50:51.905 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.904+0000 7f6e827fc640 1 -- 172.21.15.155:0/1044154739 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6e88059930 con 0x7f6e8c0b22e0 2024-08-09T14:50:51.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.904+0000 7f6e827fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-09T14:50:51.906 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.904+0000 7f6e827fc640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:51.906 INFO:teuthology.orchestra.run.smithi155.stdout:90194313225 2024-08-09T14:50:51.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 10 librados: watch_flush enter 2024-08-09T14:50:51.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 10 librados: watch_flush exit 2024-08-09T14:50:51.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 20 client.4310.objecter shutdown clearing up homeless session... 2024-08-09T14:50:51.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 10 client.4310.objecter successfully canceled tick 2024-08-09T14:50:51.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 1 -- 172.21.15.155:0/1044154739 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f6e7006d7c0 msgr2=0x7f6e7006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 1 --2- 172.21.15.155:0/1044154739 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f6e7006d7c0 0x7f6e7006fc80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c035000 tx=0x7f6e7c030000 comp rx=0 tx=0).stop 2024-08-09T14:50:51.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 10 monclient: shutdown 2024-08-09T14:50:51.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:51.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 1 -- 172.21.15.155:0/1044154739 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e8c0b22e0 msgr2=0x7f6e8c06fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:51.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 1 --2- 172.21.15.155:0/1044154739 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e8c0b22e0 0x7f6e8c06fe00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800e9b0 tx=0x7f6e8800ee70 comp rx=0 tx=0).stop 2024-08-09T14:50:51.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 1 -- 172.21.15.155:0/1044154739 shutdown_connections 2024-08-09T14:50:51.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 1 --2- 172.21.15.155:0/1044154739 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f6e7006d7c0 0x7f6e7006fc80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 1 --2- 172.21.15.155:0/1044154739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e8c146f10 0x7f6e8c070340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 1 --2- 172.21.15.155:0/1044154739 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f6e8c0b22e0 0x7f6e8c06fe00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 1 --2- 172.21.15.155:0/1044154739 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f6e8c0b0160 0x7f6e8c072750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:51.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 1 -- 172.21.15.155:0/1044154739 >> 172.21.15.155:0/1044154739 conn(0x7f6e8c06dbd0 msgr2=0x7f6e8c143200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:51.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 1 -- 172.21.15.155:0/1044154739 shutdown_connections 2024-08-09T14:50:51.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 1 -- 172.21.15.155:0/1044154739 wait complete. 2024-08-09T14:50:51.910 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:51.906+0000 7f6e93faf640 1 librados: shutdown 2024-08-09T14:50:51.920 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313227 got 90194313225 for osd.0 2024-08-09T14:50:52.449 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-08-09T14:50:52.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.549+0000 7f4d42b2e640 10 monclient: get_monmap_and_config 2024-08-09T14:50:52.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.549+0000 7f4d42b2e640 10 monclient: build_initial_monmap 2024-08-09T14:50:52.550 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.549+0000 7f4d42b2e640 10 monclient: monmap: 2024-08-09T14:50:52.550 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:52.550 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.551 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:52.550197+0000 2024-08-09T14:50:52.551 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:52.550197+0000 2024-08-09T14:50:52.551 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:52.551 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.551 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:52.551 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:52.551 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:52.551 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:52.551 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:52.551 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.549+0000 7f4d42b2e640 1 Processor -- start 2024-08-09T14:50:52.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.549+0000 7f4d42b2e640 1 -- start start 2024-08-09T14:50:52.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.549+0000 7f4d42b2e640 10 monclient: init 2024-08-09T14:50:52.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.550+0000 7f4d42b2e640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:52.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.550+0000 7f4d42b2e640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-08-09T14:50:52.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.550+0000 7f4d42b2e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d3c0b0160 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:52.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.550+0000 7f4d42b2e640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4d3c0b0540 0x7f4d3c1469d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.550+0000 7f4d42b2e640 10 monclient(hunting): picked mon.noname-a con 0x7f4d3c0b0540 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:52.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.550+0000 7f4d42b2e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d3c146f10 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:52.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.550+0000 7f4d42b2e640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d3c1461c0 con 0x7f4d3c0b0160 2024-08-09T14:50:52.553 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.550+0000 7f4d42b2e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d3c0b3f50 con 0x7f4d3c146f10 2024-08-09T14:50:52.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.550+0000 7f4d42b2e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.550+0000 7f4d42b2e640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f4d3c0b4120 con 0x7f4d3c0b0540 2024-08-09T14:50:52.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.550+0000 7f4d42b2e640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:52.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.550+0000 7f4d42b2e640 10 monclient(hunting): authenticate will time out at 1259.587630s 2024-08-09T14:50:52.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.551+0000 7f4d3bfff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f4d3c0b0540 0x7f4d3c1469d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.551+0000 7f4d410cb640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f4d3c146f10 0x7f4d3c0b3840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.155:40220/0 (socket says 172.21.15.155:40220) 2024-08-09T14:50:52.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.551+0000 7f4d410cb640 1 -- 172.21.15.155:0/1085834515 learned_addr learned my addr 172.21.15.155:0/1085834515 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:52.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.551+0000 7f4d3bfff640 10 monclient(hunting): get_auth_request con 0x7f4d3c0b0540 auth_method 0 2024-08-09T14:50:52.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.551+0000 7f4d3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:52.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.551+0000 7f4d3bfff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.551+0000 7f4d3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.552+0000 7f4d3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:52.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.552+0000 7f4d3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:52.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.552+0000 7f4d3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:52.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.554+0000 7f4d3b7fe640 1 -- 172.21.15.155:0/1085834515 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 529710801 0 0) 0x7f4d3c0b3f50 con 0x7f4d3c146f10 2024-08-09T14:50:52.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.554+0000 7f4d3b7fe640 10 monclient(hunting): _negotiate 2024-08-09T14:50:52.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.554+0000 7f4d3b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.554+0000 7f4d3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.554+0000 7f4d3b7fe640 10 monclient(hunting): my global_id is 4339 2024-08-09T14:50:52.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.554+0000 7f4d3b7fe640 1 -- 172.21.15.155:0/1085834515 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d24003520 con 0x7f4d3c146f10 2024-08-09T14:50:52.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.554+0000 7f4d3bfff640 10 monclient(hunting): handle_auth_done global_id 4356 payload 274 2024-08-09T14:50:52.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.554+0000 7f4d3bfff640 1 -- 172.21.15.155:0/1085834515 >> v1:172.21.15.194:6789/0 conn(0x7f4d3c146f10 legacy=0x7f4d3c0b3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.554+0000 7f4d3bfff640 1 -- 172.21.15.155:0/1085834515 >> v1:172.21.15.155:6789/0 conn(0x7f4d3c0b0160 legacy=0x7f4d3c0b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.554+0000 7f4d3bfff640 10 monclient: _finish_hunting 0 2024-08-09T14:50:52.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.554+0000 7f4d3bfff640 1 monclient: found mon.noname-a 2024-08-09T14:50:52.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.554+0000 7f4d3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:52.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.554+0000 7f4d3bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:52.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.554+0000 7f4d3bfff640 1 -- 172.21.15.155:0/1085834515 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3c0b42a0 con 0x7f4d3c0b0540 2024-08-09T14:50:52.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.557+0000 7f4d3bfff640 1 --2- 172.21.15.155:0/1085834515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d3c0b0540 0x7f4d3c1469d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4d300097a0 tx=0x7f4d3002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=213625274eaa57a3 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:52.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d3b7fe640 1 -- 172.21.15.155:0/1085834515 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d300315a0 con 0x7f4d3c0b0540 2024-08-09T14:50:52.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d3b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:52.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d3b7fe640 10 monclient: dump: 2024-08-09T14:50:52.564 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.564 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.565 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.565 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.565 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.565 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.565 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.565 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.565 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.565 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d3b7fe640 10 monclient: _finish_auth 0 2024-08-09T14:50:52.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d3b7fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:52.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:52.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d3b7fe640 1 -- 172.21.15.155:0/1085834515 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d30037070 con 0x7f4d3c0b0540 2024-08-09T14:50:52.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d3b7fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:52.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d3b7fe640 1 -- 172.21.15.155:0/1085834515 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d30031960 con 0x7f4d3c0b0540 2024-08-09T14:50:52.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d42b2e640 5 monclient: authenticate success, global_id 4356 2024-08-09T14:50:52.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d42b2e640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:52.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d42b2e640 4 set_mon_vals no callback set 2024-08-09T14:50:52.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d3b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:52.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d3b7fe640 10 monclient: dump: 2024-08-09T14:50:52.567 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.567 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.567 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.567 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.567 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.567 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.567 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.567 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.567 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.567 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d42b2e640 10 monclient: shutdown 2024-08-09T14:50:52.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d42b2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:52.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d42b2e640 1 -- 172.21.15.155:0/1085834515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d3c0b0540 msgr2=0x7f4d3c1469d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.563+0000 7f4d42b2e640 1 --2- 172.21.15.155:0/1085834515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d3c0b0540 0x7f4d3c1469d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4d300097a0 tx=0x7f4d3002b840 comp rx=0 tx=0).stop 2024-08-09T14:50:52.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.564+0000 7f4d42b2e640 1 -- 172.21.15.155:0/1085834515 shutdown_connections 2024-08-09T14:50:52.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.564+0000 7f4d42b2e640 1 --2- 172.21.15.155:0/1085834515 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d3c0b0540 0x7f4d3c1469d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.564+0000 7f4d42b2e640 1 -- 172.21.15.155:0/1085834515 >> 172.21.15.155:0/1085834515 conn(0x7f4d3c06dbd0 msgr2=0x7f4d3c0676f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:52.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.564+0000 7f4d42b2e640 1 -- 172.21.15.155:0/1085834515 shutdown_connections 2024-08-09T14:50:52.572 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.564+0000 7f4d42b2e640 1 -- 172.21.15.155:0/1085834515 wait complete. 2024-08-09T14:50:52.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.564+0000 7f4d42b2e640 10 monclient: build_initial_monmap 2024-08-09T14:50:52.573 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.564+0000 7f4d42b2e640 10 monclient: monmap: 2024-08-09T14:50:52.573 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:52.573 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:52.573 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:52.573 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:52.573 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:52.573 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.573 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:52.573 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:52.573 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:52.573 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.565+0000 7f4d42b2e640 1 librados: starting msgr at 2024-08-09T14:50:52.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.565+0000 7f4d42b2e640 1 librados: starting objecter 2024-08-09T14:50:52.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.565+0000 7f4d42b2e640 1 Processor -- start 2024-08-09T14:50:52.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.565+0000 7f4d42b2e640 1 -- start start 2024-08-09T14:50:52.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.565+0000 7f4d42b2e640 1 librados: setting wanted keys 2024-08-09T14:50:52.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.565+0000 7f4d42b2e640 1 librados: calling monclient init 2024-08-09T14:50:52.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.565+0000 7f4d42b2e640 10 monclient: init 2024-08-09T14:50:52.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.565+0000 7f4d42b2e640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:52.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 10 monclient: _add_conns ranks=[2,1,0] 2024-08-09T14:50:52.574 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4d3c0b0160 0x7f4d3c145580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 10 monclient(hunting): picked mon.noname-c con 0x7f4d3c0b0160 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:52.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d3c0b0540 0x7f4d3c13e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 10 monclient(hunting): picked mon.noname-b con 0x7f4d3c0b0540 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:52.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d3c146f10 0x7f4d3c13e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 10 monclient(hunting): picked mon.noname-a con 0x7f4d3c146f10 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:52.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c14e8f0 con 0x7f4d3c146f10 2024-08-09T14:50:52.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c14e770 con 0x7f4d3c0b0540 2024-08-09T14:50:52.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f4d3c14ea70 con 0x7f4d3c0b0160 2024-08-09T14:50:52.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:52.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d42b2e640 10 monclient(hunting): authenticate will time out at 1259.603066s 2024-08-09T14:50:52.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d410cb640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d3c146f10 0x7f4d3c13e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d3bfff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d3c0b0540 0x7f4d3c13e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d408ca640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4d3c0b0160 0x7f4d3c145580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d3bfff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d3c0b0540 0x7f4d3c13e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:47994/0 (socket says 172.21.15.155:47994) 2024-08-09T14:50:52.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d408ca640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4d3c0b0160 0x7f4d3c145580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54816/0 (socket says 172.21.15.155:54816) 2024-08-09T14:50:52.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d3bfff640 1 -- 172.21.15.155:0/3868600138 learned_addr learned my addr 172.21.15.155:0/3868600138 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:52.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d3bfff640 10 monclient(hunting): get_auth_request con 0x7f4d3c0b0540 auth_method 0 2024-08-09T14:50:52.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:52.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d3bfff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d408ca640 10 monclient(hunting): get_auth_request con 0x7f4d3c0b0160 auth_method 0 2024-08-09T14:50:52.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d408ca640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:52.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d408ca640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d408ca640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d410cb640 10 monclient(hunting): get_auth_request con 0x7f4d3c146f10 auth_method 0 2024-08-09T14:50:52.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d410cb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:52.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d410cb640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.566+0000 7f4d410cb640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.567+0000 7f4d3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:52.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.567+0000 7f4d3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:52.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.567+0000 7f4d3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:52.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.567+0000 7f4d408ca640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:52.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.567+0000 7f4d408ca640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:52.577 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.567+0000 7f4d408ca640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:52.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.567+0000 7f4d410cb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:52.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.567+0000 7f4d410cb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:52.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.567+0000 7f4d410cb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:52.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.570+0000 7f4d3bfff640 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2024-08-09T14:50:52.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.570+0000 7f4d3bfff640 1 -- 172.21.15.155:0/3868600138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4d3c0b0160 msgr2=0x7f4d3c145580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.570+0000 7f4d3bfff640 1 --2- 172.21.15.155:0/3868600138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4d3c0b0160 0x7f4d3c145580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.570+0000 7f4d3bfff640 1 -- 172.21.15.155:0/3868600138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d3c146f10 msgr2=0x7f4d3c13e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.570+0000 7f4d3bfff640 1 --2- 172.21.15.155:0/3868600138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d3c146f10 0x7f4d3c13e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.570+0000 7f4d3bfff640 10 monclient: _finish_hunting 0 2024-08-09T14:50:52.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.570+0000 7f4d3bfff640 1 monclient: found mon.noname-b 2024-08-09T14:50:52.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.570+0000 7f4d3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:52.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.570+0000 7f4d3bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:52.578 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.570+0000 7f4d3bfff640 1 -- 172.21.15.155:0/3868600138 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3c13de10 con 0x7f4d3c0b0540 2024-08-09T14:50:52.580 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.572+0000 7f4d3bfff640 1 --2- 172.21.15.155:0/3868600138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d3c0b0540 0x7f4d3c13e2f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4d30034040 tx=0x7f4d300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:52.580 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.572+0000 7f4d397fa640 10 client.?.objecter ms_handle_connect 0x7f4d3c0b0540 2024-08-09T14:50:52.581 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.572+0000 7f4d397fa640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:52.581 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d397fa640 1 -- 172.21.15.155:0/3868600138 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d30032630 con 0x7f4d3c0b0540 2024-08-09T14:50:52.581 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.581 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d397fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:52.581 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d397fa640 10 monclient: dump: 2024-08-09T14:50:52.581 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.581 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.581 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.581 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.581 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.581 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.581 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.582 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.582 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.582 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d397fa640 10 monclient: _finish_auth 0 2024-08-09T14:50:52.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d397fa640 10 monclient: _check_auth_tickets 2024-08-09T14:50:52.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:52.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d397fa640 1 -- 172.21.15.155:0/3868600138 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4d300327f0 con 0x7f4d3c0b0540 2024-08-09T14:50:52.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d397fa640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:52.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d397fa640 1 -- 172.21.15.155:0/3868600138 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d30032be0 con 0x7f4d3c0b0540 2024-08-09T14:50:52.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d42b2e640 5 monclient: authenticate success, global_id 4342 2024-08-09T14:50:52.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d42b2e640 10 monclient: _renew_subs 2024-08-09T14:50:52.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d42b2e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:52.582 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d42b2e640 1 -- 172.21.15.155:0/3868600138 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3c13fdb0 con 0x7f4d3c0b0540 2024-08-09T14:50:52.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d397fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d397fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:52.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d397fa640 10 monclient: dump: 2024-08-09T14:50:52.583 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.583 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.583 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.583 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.583 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.583 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.583 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.583 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.583 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.583 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.583 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d3affd640 4 set_mon_vals no callback set 2024-08-09T14:50:52.584 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d42b2e640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:52.584 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d42b2e640 10 monclient: _renew_subs 2024-08-09T14:50:52.584 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d42b2e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:52.584 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d42b2e640 1 -- 172.21.15.155:0/3868600138 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d3c13da30 con 0x7f4d3c0b0540 2024-08-09T14:50:52.584 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.576+0000 7f4d42b2e640 1 librados: init done 2024-08-09T14:50:52.584 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.577+0000 7f4d42b2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:52.584 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.577+0000 7f4d42b2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:52.584 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.577+0000 7f4d42b2e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:52.584 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.577+0000 7f4d42b2e640 1 -- 172.21.15.155:0/3868600138 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d3c1469d0 con 0x7f4d3c0b0540 2024-08-09T14:50:52.610 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-08-09T14:50:52.617 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-08-09T14:50:52.678 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-08-09T14:50:52.682 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-09T14:50:52.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.718+0000 7f100d08a640 10 monclient: get_monmap_and_config 2024-08-09T14:50:52.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.718+0000 7f100d08a640 10 monclient: build_initial_monmap 2024-08-09T14:50:52.720 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.718+0000 7f100d08a640 10 monclient: monmap: 2024-08-09T14:50:52.720 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:52.720 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.721 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:52.719517+0000 2024-08-09T14:50:52.721 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:52.719517+0000 2024-08-09T14:50:52.721 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:52.721 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.721 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:52.721 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:52.721 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:52.721 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:52.721 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:52.721 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.719+0000 7f100d08a640 1 Processor -- start 2024-08-09T14:50:52.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.719+0000 7f100d08a640 1 -- start start 2024-08-09T14:50:52.721 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.719+0000 7f100d08a640 10 monclient: init 2024-08-09T14:50:52.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.719+0000 7f100d08a640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:52.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.719+0000 7f100d08a640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-08-09T14:50:52.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.719+0000 7f100d08a640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f10080b0160 0x7f10080b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.723 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.719+0000 7f100d08a640 10 monclient(hunting): picked mon.noname-a con 0x7f10080b0160 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:52.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.720+0000 7f100d08a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10080b22e0 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:52.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.720+0000 7f100d08a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10080b26e0 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:52.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.720+0000 7f100d08a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10081470e0 con 0x7f10080b22e0 2024-08-09T14:50:52.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.720+0000 7f100d08a640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10081461c0 con 0x7f10080b26e0 2024-08-09T14:50:52.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.720+0000 7f100d08a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.720+0000 7f100d08a640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f10080b3f50 con 0x7f10080b0160 2024-08-09T14:50:52.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.720+0000 7f100d08a640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:52.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.720+0000 7f100d08a640 10 monclient(hunting): authenticate will time out at 1259.757142s 2024-08-09T14:50:52.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.720+0000 7f1006d9d640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f10080b0160 0x7f10080b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.721+0000 7f1006d9d640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f10080b0160 0x7f10080b0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43674/0 (socket says 172.21.15.155:43674) 2024-08-09T14:50:52.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.721+0000 7f1006d9d640 1 -- 172.21.15.155:0/1860480462 learned_addr learned my addr 172.21.15.155:0/1860480462 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:52.724 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.721+0000 7f1006d9d640 10 monclient(hunting): get_auth_request con 0x7f10080b0160 auth_method 0 2024-08-09T14:50:52.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.721+0000 7f1006d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:52.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.721+0000 7f1006d9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.725 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.721+0000 7f1006d9d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.726 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.724+0000 7f1006d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:52.726 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.724+0000 7f1006d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:52.726 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.724+0000 7f1006d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:52.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.727+0000 7f1006d9d640 10 monclient(hunting): handle_auth_done global_id 4365 payload 274 2024-08-09T14:50:52.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.728+0000 7f1006d9d640 1 -- 172.21.15.155:0/1860480462 >> v1:172.21.15.194:6789/0 conn(0x7f10080b26e0 legacy=0x7f10080b3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:52.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.728+0000 7f1006d9d640 1 -- 172.21.15.155:0/1860480462 >> v1:172.21.15.155:6789/0 conn(0x7f10080b22e0 legacy=0x7f10081469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:52.729 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.728+0000 7f1006d9d640 10 monclient: _finish_hunting 0 2024-08-09T14:50:52.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.728+0000 7f1006d9d640 1 monclient: found mon.noname-a 2024-08-09T14:50:52.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.728+0000 7f1006d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:52.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.728+0000 7f1006d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:52.730 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.728+0000 7f1006d9d640 1 -- 172.21.15.155:0/1860480462 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10080b40d0 con 0x7f10080b0160 2024-08-09T14:50:52.733 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.731+0000 7f1006d9d640 1 --2- 172.21.15.155:0/1860480462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10080b0160 0x7f10080b0540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009870 tx=0x7f0ff802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f51904f226c7117f server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:52.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.740+0000 7f1005d9b640 1 -- 172.21.15.155:0/1860480462 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8038070 con 0x7f10080b0160 2024-08-09T14:50:52.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.740+0000 7f1005d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.740+0000 7f1005d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:52.743 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.740+0000 7f1005d9b640 10 monclient: dump: 2024-08-09T14:50:52.743 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.744 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.744 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.744 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.744 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.744 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.744 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.745 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.745 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.745 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.740+0000 7f1005d9b640 10 monclient: _finish_auth 0 2024-08-09T14:50:52.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.740+0000 7f1005d9b640 10 monclient: _check_auth_tickets 2024-08-09T14:50:52.745 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.740+0000 7f1005d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:52.746 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.740+0000 7f100d08a640 5 monclient: authenticate success, global_id 4365 2024-08-09T14:50:52.746 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.740+0000 7f100d08a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:52.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.743+0000 7f1005d9b640 1 -- 172.21.15.155:0/1860480462 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ff802f050 con 0x7f10080b0160 2024-08-09T14:50:52.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.743+0000 7f1005d9b640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:52.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.743+0000 7f1005d9b640 1 -- 172.21.15.155:0/1860480462 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8031970 con 0x7f10080b0160 2024-08-09T14:50:52.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.743+0000 7f1005d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.743+0000 7f1005d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:52.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.743+0000 7f1005d9b640 10 monclient: dump: 2024-08-09T14:50:52.756 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.756 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.756 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.757 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.757 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.757 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.757 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.757 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.757 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.757 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.743+0000 7f100d08a640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:52.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.743+0000 7f100d08a640 4 set_mon_vals no callback set 2024-08-09T14:50:52.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.744+0000 7f100d08a640 10 monclient: shutdown 2024-08-09T14:50:52.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.744+0000 7f100d08a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:52.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.744+0000 7f100d08a640 1 -- 172.21.15.155:0/1860480462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10080b0160 msgr2=0x7f10080b0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.744+0000 7f100d08a640 1 --2- 172.21.15.155:0/1860480462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10080b0160 0x7f10080b0540 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8009870 tx=0x7f0ff802b7f0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.744+0000 7f100d08a640 1 -- 172.21.15.155:0/1860480462 shutdown_connections 2024-08-09T14:50:52.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.744+0000 7f100d08a640 1 --2- 172.21.15.155:0/1860480462 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10080b0160 0x7f10080b0540 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.744+0000 7f100d08a640 1 -- 172.21.15.155:0/1860480462 >> 172.21.15.155:0/1860480462 conn(0x7f100806dbd0 msgr2=0x7f10080676f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:52.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.744+0000 7f100d08a640 1 -- 172.21.15.155:0/1860480462 shutdown_connections 2024-08-09T14:50:52.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.744+0000 7f100d08a640 1 -- 172.21.15.155:0/1860480462 wait complete. 2024-08-09T14:50:52.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.746+0000 7f100d08a640 10 monclient: build_initial_monmap 2024-08-09T14:50:52.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.746+0000 7f100d08a640 10 monclient: monmap: 2024-08-09T14:50:52.758 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:52.758 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:52.758 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:52.758 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:52.758 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:52.759 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.759 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:52.759 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:52.759 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:52.759 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.746+0000 7f100d08a640 1 librados: starting msgr at 2024-08-09T14:50:52.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.746+0000 7f100d08a640 1 librados: starting objecter 2024-08-09T14:50:52.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.747+0000 7f100d08a640 1 Processor -- start 2024-08-09T14:50:52.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.747+0000 7f100d08a640 1 -- start start 2024-08-09T14:50:52.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.747+0000 7f100d08a640 1 librados: setting wanted keys 2024-08-09T14:50:52.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.747+0000 7f100d08a640 1 librados: calling monclient init 2024-08-09T14:50:52.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.747+0000 7f100d08a640 10 monclient: init 2024-08-09T14:50:52.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.748+0000 7f100d08a640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:52.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.748+0000 7f100d08a640 10 monclient: _add_conns ranks=[2,0,1] 2024-08-09T14:50:52.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.748+0000 7f100d08a640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10080b0160 0x7f1008142950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.748+0000 7f100d08a640 10 monclient(hunting): picked mon.noname-c con 0x7f10080b0160 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:52.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.748+0000 7f100d08a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10080b22e0 0x7f1008141e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.748+0000 7f100d08a640 10 monclient(hunting): picked mon.noname-a con 0x7f10080b22e0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:52.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.748+0000 7f100d08a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10080b26e0 0x7f1008134710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.748+0000 7f100d08a640 10 monclient(hunting): picked mon.noname-b con 0x7f10080b26e0 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:52.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.748+0000 7f100d08a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.748+0000 7f100d08a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f100814e8f0 con 0x7f10080b22e0 2024-08-09T14:50:52.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.748+0000 7f100d08a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.748+0000 7f100d08a640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f100814e770 con 0x7f10080b26e0 2024-08-09T14:50:52.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.748+0000 7f100d08a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.748+0000 7f100d08a640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f100814ea70 con 0x7f10080b0160 2024-08-09T14:50:52.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.749+0000 7f100d08a640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:52.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.749+0000 7f100d08a640 10 monclient(hunting): authenticate will time out at 1259.786070s 2024-08-09T14:50:52.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.749+0000 7f100659c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10080b22e0 0x7f1008141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.749+0000 7f1006d9d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10080b0160 0x7f1008142950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.749+0000 7f100659c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10080b22e0 0x7f1008141e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43686/0 (socket says 172.21.15.155:43686) 2024-08-09T14:50:52.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.749+0000 7f100659c640 1 -- 172.21.15.155:0/3040544002 learned_addr learned my addr 172.21.15.155:0/3040544002 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:52.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.749+0000 7f100659c640 10 monclient(hunting): get_auth_request con 0x7f10080b22e0 auth_method 0 2024-08-09T14:50:52.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.749+0000 7f100659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:52.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.749+0000 7f100659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.749+0000 7f100659c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.749+0000 7f1006d9d640 10 monclient(hunting): get_auth_request con 0x7f10080b0160 auth_method 0 2024-08-09T14:50:52.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.749+0000 7f1006d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:52.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.749+0000 7f1006d9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.749+0000 7f1006d9d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.750+0000 7f100659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:52.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.750+0000 7f100659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:52.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.750+0000 7f100659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:52.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.750+0000 7f1006d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:52.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.750+0000 7f1006d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:52.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.750+0000 7f1006d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:52.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.753+0000 7f100659c640 10 monclient(hunting): handle_auth_done global_id 4368 payload 824 2024-08-09T14:50:52.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.753+0000 7f100659c640 1 -- 172.21.15.155:0/3040544002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10080b0160 msgr2=0x7f1008142950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.753+0000 7f100659c640 1 --2- 172.21.15.155:0/3040544002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10080b0160 0x7f1008142950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.753+0000 7f100659c640 1 -- 172.21.15.155:0/3040544002 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10080b26e0 msgr2=0x7f1008134710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.753+0000 7f100659c640 1 --2- 172.21.15.155:0/3040544002 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10080b26e0 0x7f1008134710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.753+0000 7f100659c640 10 monclient: _finish_hunting 0 2024-08-09T14:50:52.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.753+0000 7f100659c640 1 monclient: found mon.noname-a 2024-08-09T14:50:52.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.754+0000 7f100659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:52.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.754+0000 7f100659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:52.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.754+0000 7f100659c640 1 -- 172.21.15.155:0/3040544002 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1008143760 con 0x7f10080b22e0 2024-08-09T14:50:52.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.757+0000 7f100659c640 1 --2- 172.21.15.155:0/3040544002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10080b22e0 0x7f1008141e10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00c430 tx=0x7f0ffc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:52.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.757+0000 7f0fef7fe640 10 client.?.objecter ms_handle_connect 0x7f10080b22e0 2024-08-09T14:50:52.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.757+0000 7f0fef7fe640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:52.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f0fef7fe640 1 -- 172.21.15.155:0/3040544002 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc011790 con 0x7f10080b22e0 2024-08-09T14:50:52.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f0fef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f0fef7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:52.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f0fef7fe640 10 monclient: dump: 2024-08-09T14:50:52.765 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.765 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.765 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.765 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.765 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.765 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.765 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.765 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.766 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.766 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f0fef7fe640 10 monclient: _finish_auth 0 2024-08-09T14:50:52.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f0fef7fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:52.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f0fef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:52.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f0fef7fe640 1 -- 172.21.15.155:0/3040544002 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ffc017070 con 0x7f10080b22e0 2024-08-09T14:50:52.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f0fef7fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:52.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f0fef7fe640 1 -- 172.21.15.155:0/3040544002 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ffc011bb0 con 0x7f10080b22e0 2024-08-09T14:50:52.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f0fef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f0fef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:52.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f0fef7fe640 10 monclient: dump: 2024-08-09T14:50:52.766 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.767 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.767 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.767 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.767 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.767 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.767 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.767 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.767 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.767 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f100d08a640 5 monclient: authenticate success, global_id 4368 2024-08-09T14:50:52.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f100559a640 4 set_mon_vals no callback set 2024-08-09T14:50:52.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f100d08a640 10 monclient: _renew_subs 2024-08-09T14:50:52.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f100d08a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:52.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f100d08a640 1 -- 172.21.15.155:0/3040544002 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1008143a40 con 0x7f10080b22e0 2024-08-09T14:50:52.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f100d08a640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:52.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f100d08a640 10 monclient: _renew_subs 2024-08-09T14:50:52.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f100d08a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:52.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f100d08a640 1 -- 172.21.15.155:0/3040544002 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1008140bb0 con 0x7f10080b22e0 2024-08-09T14:50:52.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.763+0000 7f100d08a640 1 librados: init done 2024-08-09T14:50:52.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.764+0000 7f100d08a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:52.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.764+0000 7f100d08a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:52.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.764+0000 7f100d08a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:52.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.764+0000 7f100d08a640 1 -- 172.21.15.155:0/3040544002 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10080b0540 con 0x7f10080b22e0 2024-08-09T14:50:52.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.770+0000 7f3fb3f67640 10 monclient: get_monmap_and_config 2024-08-09T14:50:52.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.770+0000 7f3fb3f67640 10 monclient: build_initial_monmap 2024-08-09T14:50:52.771 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.770+0000 7f3fb3f67640 10 monclient: monmap: 2024-08-09T14:50:52.771 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:52.771 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.771 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:52.771116+0000 2024-08-09T14:50:52.771 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:52.771116+0000 2024-08-09T14:50:52.771 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:52.771 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.771 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:52.771 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:52.772 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:52.772 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:52.772 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:52.772 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.770+0000 7f3fb3f67640 1 Processor -- start 2024-08-09T14:50:52.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.771+0000 7f3fb3f67640 1 -- start start 2024-08-09T14:50:52.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.771+0000 7f3fb3f67640 10 monclient: init 2024-08-09T14:50:52.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.771+0000 7f3fb3f67640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:52.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.771+0000 7f3fb3f67640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-08-09T14:50:52.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.771+0000 7f3fb3f67640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3fac0b0160 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:52.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.771+0000 7f3fb3f67640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3fac0b0540 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:52.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.771+0000 7f3fb3f67640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f3fac0b3840 0x7f3fac0b3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.771+0000 7f3fb3f67640 10 monclient(hunting): picked mon.noname-c con 0x7f3fac0b3840 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:52.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.771+0000 7f3fb3f67640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fac1470e0 con 0x7f3fac0b0160 2024-08-09T14:50:52.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.771+0000 7f3fb3f67640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3fac1461c0 con 0x7f3fac0b0540 2024-08-09T14:50:52.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.771+0000 7f3fb3f67640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.771+0000 7f3fb3f67640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f3fac0b4210 con 0x7f3fac0b3840 2024-08-09T14:50:52.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.771+0000 7f3fb3f67640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:52.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.771+0000 7f3fb3f67640 10 monclient(hunting): authenticate will time out at 1259.808694s 2024-08-09T14:50:52.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.790+0000 7f0fef7fe640 1 -- 172.21.15.155:0/3040544002 <== mon.0 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f0ffc01c460 con 0x7f10080b22e0 2024-08-09T14:50:52.791 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.790+0000 7f0fef7fe640 1 --2- 172.21.15.155:0/3040544002 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f0ff406d7c0 0x7f0ff406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.790+0000 7f1006d9d640 1 --2- 172.21.15.155:0/3040544002 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f0ff406d7c0 0x7f0ff406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.790+0000 7f1006d9d640 10 monclient: get_auth_request con 0x7f0ff406d7c0 auth_method 0 2024-08-09T14:50:52.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.791+0000 7f0fef7fe640 10 client.4368.objecter ms_handle_connect 0x7f0ff406d7c0 2024-08-09T14:50:52.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.791+0000 7f1006d9d640 1 --2- 172.21.15.155:0/3040544002 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f0ff406d7c0 0x7f0ff406fc80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8035000 tx=0x7f0ff8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:52.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.794+0000 7f0fef7fe640 1 -- 172.21.15.155:0/3040544002 <== mon.0 v2:172.21.15.155:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f0ffc057b40 con 0x7f10080b22e0 2024-08-09T14:50:52.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.794+0000 7f0fef7fe640 10 client.4368.objecter ms_dispatch 0x7f1008000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:52.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.794+0000 7f0fef7fe640 3 client.4368.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:52.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.794+0000 7f0fef7fe640 3 client.4368.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:52.796 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.794+0000 7f0fef7fe640 20 client.4368.objecter dump_active .. 0 homeless 2024-08-09T14:50:52.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.808+0000 7f819d913640 10 monclient: get_monmap_and_config 2024-08-09T14:50:52.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.808+0000 7f819d913640 10 monclient: build_initial_monmap 2024-08-09T14:50:52.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.808+0000 7f819d913640 10 monclient: monmap: 2024-08-09T14:50:52.810 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:52.810 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.810 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:52.809449+0000 2024-08-09T14:50:52.810 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:52.809449+0000 2024-08-09T14:50:52.810 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:52.810 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.810 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:52.810 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:52.810 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:52.810 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:52.810 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:52.810 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 1 Processor -- start 2024-08-09T14:50:52.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 1 -- start start 2024-08-09T14:50:52.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 10 monclient: init 2024-08-09T14:50:52.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:52.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-08-09T14:50:52.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f81980b0160 0x7f81980b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 10 monclient(hunting): picked mon.noname-c con 0x7f81980b0160 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:52.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81980b22e0 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:52.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81980b26e0 0x7f81980b3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 10 monclient(hunting): picked mon.noname-b con 0x7f81980b26e0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:52.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81981470e0 con 0x7f81980b22e0 2024-08-09T14:50:52.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f81981461c0 con 0x7f81980b0160 2024-08-09T14:50:52.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f81980b3d80 con 0x7f81980b26e0 2024-08-09T14:50:52.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:52.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.809+0000 7f819d913640 10 monclient(hunting): authenticate will time out at 1259.846957s 2024-08-09T14:50:52.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.810+0000 7f81977fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81980b26e0 0x7f81980b3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.810+0000 7f81977fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81980b26e0 0x7f81980b3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54828/0 (socket says 172.21.15.155:54828) 2024-08-09T14:50:52.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.810+0000 7f81977fe640 1 -- 172.21.15.155:0/4027357913 learned_addr learned my addr 172.21.15.155:0/4027357913 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:52.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.810+0000 7f81977fe640 10 monclient(hunting): get_auth_request con 0x7f81980b26e0 auth_method 0 2024-08-09T14:50:52.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.810+0000 7f81977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:52.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.810+0000 7f81977fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.810+0000 7f81977fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.811+0000 7f81977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:52.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.811+0000 7f81977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:52.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.811+0000 7f81977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:52.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.812+0000 7f81977fe640 10 monclient(hunting): handle_auth_done global_id 4319 payload 274 2024-08-09T14:50:52.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.813+0000 7f81977fe640 1 -- 172.21.15.155:0/4027357913 >> v2:172.21.15.194:3300/0 conn(0x7f81980b0160 msgr2=0x7f81980b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:52.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.813+0000 7f81977fe640 1 --2- 172.21.15.155:0/4027357913 >> v2:172.21.15.194:3300/0 conn(0x7f81980b0160 0x7f81980b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.813+0000 7f81977fe640 1 -- 172.21.15.155:0/4027357913 >> v1:172.21.15.194:6789/0 conn(0x7f81980b22e0 legacy=0x7f81981469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:52.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.813+0000 7f81977fe640 10 monclient: _finish_hunting 0 2024-08-09T14:50:52.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.813+0000 7f81977fe640 1 monclient: found mon.noname-b 2024-08-09T14:50:52.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.813+0000 7f81977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:52.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.813+0000 7f81977fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:52.815 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.813+0000 7f81977fe640 1 -- 172.21.15.155:0/4027357913 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81980b3f30 con 0x7f81980b26e0 2024-08-09T14:50:52.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.815+0000 7f81977fe640 1 --2- 172.21.15.155:0/4027357913 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81980b26e0 0x7f81980b3840 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8190009e50 tx=0x7f819002bdb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f059daab74e2c0c0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:52.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.817+0000 7f81967fc640 1 -- 172.21.15.155:0/4027357913 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f819002c950 con 0x7f81980b26e0 2024-08-09T14:50:52.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.817+0000 7f81967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.817+0000 7f81967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:52.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.817+0000 7f81967fc640 10 monclient: dump: 2024-08-09T14:50:52.819 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.819 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.819 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.819 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.819 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.819 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.819 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.819 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.819 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.819 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.817+0000 7f81967fc640 10 monclient: _finish_auth 0 2024-08-09T14:50:52.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.817+0000 7f81967fc640 10 monclient: _check_auth_tickets 2024-08-09T14:50:52.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.817+0000 7f81967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:52.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.817+0000 7f819d913640 5 monclient: authenticate success, global_id 4319 2024-08-09T14:50:52.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.817+0000 7f819d913640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:52.820 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.819+0000 7f81967fc640 1 -- 172.21.15.155:0/4027357913 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f819002caf0 con 0x7f81980b26e0 2024-08-09T14:50:52.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.819+0000 7f81967fc640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:52.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.819+0000 7f81967fc640 1 -- 172.21.15.155:0/4027357913 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81900343f0 con 0x7f81980b26e0 2024-08-09T14:50:52.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.819+0000 7f81967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.819+0000 7f81967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:52.821 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.819+0000 7f81967fc640 10 monclient: dump: 2024-08-09T14:50:52.821 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.821 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.821 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.821 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.821 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.821 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.821 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.821 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.821 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.822 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.819+0000 7f819d913640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:52.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.819+0000 7f819d913640 4 set_mon_vals no callback set 2024-08-09T14:50:52.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.819+0000 7f84d6bee640 10 monclient: get_monmap_and_config 2024-08-09T14:50:52.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.819+0000 7f84d6bee640 10 monclient: build_initial_monmap 2024-08-09T14:50:52.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.819+0000 7f84d6bee640 10 monclient: monmap: 2024-08-09T14:50:52.822 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:52.822 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.822 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:52.820869+0000 2024-08-09T14:50:52.822 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:52.820869+0000 2024-08-09T14:50:52.823 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:52.823 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.823 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:52.823 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:52.823 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:52.823 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:52.823 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:52.823 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.820+0000 7f819d913640 10 monclient: shutdown 2024-08-09T14:50:52.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.820+0000 7f819d913640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:52.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.820+0000 7f819d913640 1 -- 172.21.15.155:0/4027357913 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81980b26e0 msgr2=0x7f81980b3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.820+0000 7f819d913640 1 --2- 172.21.15.155:0/4027357913 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81980b26e0 0x7f81980b3840 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8190009e50 tx=0x7f819002bdb0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.820+0000 7f819d913640 1 -- 172.21.15.155:0/4027357913 shutdown_connections 2024-08-09T14:50:52.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.820+0000 7f819d913640 1 --2- 172.21.15.155:0/4027357913 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81980b26e0 0x7f81980b3840 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.820+0000 7f819d913640 1 --2- 172.21.15.155:0/4027357913 >> v2:172.21.15.194:3300/0 conn(0x7f81980b0160 0x7f81980b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.820+0000 7f819d913640 1 -- 172.21.15.155:0/4027357913 >> 172.21.15.155:0/4027357913 conn(0x7f819806dbd0 msgr2=0x7f81980676f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:52.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.820+0000 7f819d913640 1 -- 172.21.15.155:0/4027357913 shutdown_connections 2024-08-09T14:50:52.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.820+0000 7f84d6bee640 1 Processor -- start 2024-08-09T14:50:52.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.820+0000 7f819d913640 1 -- 172.21.15.155:0/4027357913 wait complete. 2024-08-09T14:50:52.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.820+0000 7f84d6bee640 1 -- start start 2024-08-09T14:50:52.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.820+0000 7f84d6bee640 10 monclient: init 2024-08-09T14:50:52.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.821+0000 7f819d913640 10 monclient: build_initial_monmap 2024-08-09T14:50:52.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.821+0000 7f819d913640 10 monclient: monmap: 2024-08-09T14:50:52.826 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:52.826 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:52.826 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:52.826 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:52.826 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:52.826 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.826 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:52.826 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:52.826 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:52.827 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.821+0000 7f84d6bee640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:52.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.821+0000 7f84d6bee640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-08-09T14:50:52.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.821+0000 7f84d6bee640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f84d006d8d0 0x7f84d00aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.821+0000 7f84d6bee640 10 monclient(hunting): picked mon.noname-c con 0x7f84d006d8d0 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:52.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.821+0000 7f84d6bee640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00b2d80 0x7f84d00b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.821+0000 7f819d913640 1 librados: starting msgr at 2024-08-09T14:50:52.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.821+0000 7f819d913640 1 librados: starting objecter 2024-08-09T14:50:52.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f819d913640 1 Processor -- start 2024-08-09T14:50:52.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84d6bee640 10 monclient(hunting): picked mon.noname-b con 0x7f84d00b2d80 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:52.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f4d397fa640 1 -- 172.21.15.155:0/3868600138 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f4d3004e430 con 0x7f4d3c0b0540 2024-08-09T14:50:52.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84d6bee640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f84d0142c50 0x7f84d013fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84d6bee640 10 monclient(hunting): picked mon.noname-a con 0x7f84d0142c50 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:52.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84d6bee640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84d6bee640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f84d006dcb0 con 0x7f84d0142c50 2024-08-09T14:50:52.830 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84d6bee640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84d6bee640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f84d00b4680 con 0x7f84d006d8d0 2024-08-09T14:50:52.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84d6bee640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84d6bee640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f84d00b4960 con 0x7f84d00b2d80 2024-08-09T14:50:52.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84d6bee640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:52.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84d6bee640 10 monclient(hunting): authenticate will time out at 1259.859385s 2024-08-09T14:50:52.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84cffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00b2d80 0x7f84d00b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84cffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00b2d80 0x7f84d00b4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54834/0 (socket says 172.21.15.155:54834) 2024-08-09T14:50:52.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f4d397fa640 1 --2- 172.21.15.155:0/3868600138 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f4d2806d810 0x7f4d2806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84cffff640 1 -- 172.21.15.155:0/4258283926 learned_addr learned my addr 172.21.15.155:0/4258283926 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:52.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84cffff640 10 monclient(hunting): get_auth_request con 0x7f84d00b2d80 auth_method 0 2024-08-09T14:50:52.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:52.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84cffff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f84cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f819d913640 1 -- start start 2024-08-09T14:50:52.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f819d913640 1 librados: setting wanted keys 2024-08-09T14:50:52.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f819d913640 1 librados: calling monclient init 2024-08-09T14:50:52.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.822+0000 7f819d913640 10 monclient: init 2024-08-09T14:50:52.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f4d408ca640 1 --2- 172.21.15.155:0/3868600138 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f4d2806d810 0x7f4d2806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f4d408ca640 10 monclient: get_auth_request con 0x7f4d2806d810 auth_method 0 2024-08-09T14:50:52.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:52.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 10 monclient: _add_conns ranks=[0,1,2] 2024-08-09T14:50:52.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f84cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:52.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81980b0160 0x7f819813ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f84cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:52.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f84cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:52.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 10 monclient(hunting): picked mon.noname-a con 0x7f81980b0160 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:52.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81980b22e0 0x7f8198144520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 10 monclient(hunting): picked mon.noname-b con 0x7f81980b22e0 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:52.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81980b26e0 0x7f8198144a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 10 monclient(hunting): picked mon.noname-c con 0x7f81980b26e0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:52.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f819814d6d0 con 0x7f81980b0160 2024-08-09T14:50:52.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f819814d550 con 0x7f81980b22e0 2024-08-09T14:50:52.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f819814d850 con 0x7f81980b26e0 2024-08-09T14:50:52.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:52.836 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f819d913640 10 monclient(hunting): authenticate will time out at 1259.860624s 2024-08-09T14:50:52.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f4d408ca640 1 --2- 172.21.15.155:0/3868600138 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f4d2806d810 0x7f4d2806fcd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c001780 tx=0x7f4d2c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:52.837 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.823+0000 7f4d397fa640 10 client.4342.objecter ms_handle_connect 0x7f4d2806d810 2024-08-09T14:50:52.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f84cffff640 10 monclient(hunting): handle_auth_done global_id 4322 payload 274 2024-08-09T14:50:52.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f84cffff640 1 -- 172.21.15.155:0/4258283926 >> v2:172.21.15.194:3300/0 conn(0x7f84d006d8d0 msgr2=0x7f84d00aa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:52.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f84cffff640 1 --2- 172.21.15.155:0/4258283926 >> v2:172.21.15.194:3300/0 conn(0x7f84d006d8d0 0x7f84d00aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f84cffff640 1 -- 172.21.15.155:0/4258283926 >> v2:172.21.15.155:3300/0 conn(0x7f84d0142c50 msgr2=0x7f84d013fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:52.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f84cffff640 1 --2- 172.21.15.155:0/4258283926 >> v2:172.21.15.155:3300/0 conn(0x7f84d0142c50 0x7f84d013fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f84cffff640 10 monclient: _finish_hunting 0 2024-08-09T14:50:52.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f84cffff640 1 monclient: found mon.noname-b 2024-08-09T14:50:52.838 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f84cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:52.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f84cffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:52.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f84cffff640 1 -- 172.21.15.155:0/4258283926 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d0143100 con 0x7f84d00b2d80 2024-08-09T14:50:52.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f4d397fa640 1 -- 172.21.15.155:0/3868600138 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f4d3002f070 con 0x7f4d3c0b0540 2024-08-09T14:50:52.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f4d397fa640 10 client.4342.objecter ms_dispatch 0x7f4d3c000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:52.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f4d397fa640 3 client.4342.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:52.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f4d397fa640 3 client.4342.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:52.839 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.825+0000 7f4d397fa640 20 client.4342.objecter dump_active .. 0 homeless 2024-08-09T14:50:52.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.826+0000 7f81977fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81980b26e0 0x7f8198144a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.827+0000 7f81977fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81980b26e0 0x7f8198144a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:54840/0 (socket says 172.21.15.155:54840) 2024-08-09T14:50:52.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.827+0000 7f81977fe640 1 -- 172.21.15.155:0/2390153102 learned_addr learned my addr 172.21.15.155:0/2390153102 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:52.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.827+0000 7f81977fe640 10 monclient(hunting): get_auth_request con 0x7f81980b26e0 auth_method 0 2024-08-09T14:50:52.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.827+0000 7f81977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:52.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.827+0000 7f81977fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.840 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.827+0000 7f81977fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.828+0000 7f84cffff640 1 --2- 172.21.15.155:0/4258283926 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00b2d80 0x7f84d00b4df0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f84d00b6520 tx=0x7f84bc02b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8922ca3c71beffc server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:52.841 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.828+0000 7f81977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:52.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.828+0000 7f81977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:52.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.828+0000 7f81977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:52.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.831+0000 7f81977fe640 10 monclient(hunting): handle_auth_done global_id 4325 payload 824 2024-08-09T14:50:52.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.831+0000 7f81977fe640 1 -- 172.21.15.155:0/2390153102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81980b22e0 msgr2=0x7f8198144520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.831+0000 7f81977fe640 1 --2- 172.21.15.155:0/2390153102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81980b22e0 0x7f8198144520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.831+0000 7f81977fe640 1 -- 172.21.15.155:0/2390153102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81980b0160 msgr2=0x7f819813ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.831+0000 7f81977fe640 1 --2- 172.21.15.155:0/2390153102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81980b0160 0x7f819813ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.831+0000 7f81977fe640 10 monclient: _finish_hunting 0 2024-08-09T14:50:52.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.831+0000 7f81977fe640 1 monclient: found mon.noname-c 2024-08-09T14:50:52.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.831+0000 7f81977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:52.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.831+0000 7f81977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:50:52.842 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.831+0000 7f81977fe640 1 -- 172.21.15.155:0/2390153102 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81980b7420 con 0x7f81980b26e0 2024-08-09T14:50:52.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.835+0000 7f81977fe640 1 --2- 172.21.15.155:0/2390153102 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81980b26e0 0x7f8198144a60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8190009e50 tx=0x7f8190004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:52.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.835+0000 7f8194ff9640 10 client.?.objecter ms_handle_connect 0x7f81980b26e0 2024-08-09T14:50:52.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.835+0000 7f8194ff9640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:52.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.836+0000 7f4d397fa640 1 -- 172.21.15.155:0/3868600138 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f4d300d9990 con 0x7f4d3c0b0540 2024-08-09T14:50:52.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.836+0000 7f4d397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:52.843 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.836+0000 7f4d397fa640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:52.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.837+0000 7f84cf7fe640 1 -- 172.21.15.155:0/4258283926 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc035020 con 0x7f84d00b2d80 2024-08-09T14:50:52.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.837+0000 7f84cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.837+0000 7f84cf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:52.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.837+0000 7f84cf7fe640 10 monclient: dump: 2024-08-09T14:50:52.847 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.847 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.847 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.847 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.847 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.847 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.847 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.848 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.837+0000 7f84cf7fe640 10 monclient: _finish_auth 0 2024-08-09T14:50:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.837+0000 7f84cf7fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.837+0000 7f84cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.837+0000 7f84cf7fe640 1 -- 172.21.15.155:0/4258283926 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84bc031870 con 0x7f84d00b2d80 2024-08-09T14:50:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.838+0000 7f84cf7fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.838+0000 7f84cf7fe640 1 -- 172.21.15.155:0/4258283926 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc031c40 con 0x7f84d00b2d80 2024-08-09T14:50:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.838+0000 7f84cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.838+0000 7f84cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.838+0000 7f84cf7fe640 10 monclient: dump: 2024-08-09T14:50:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.848 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.849 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.838+0000 7f84d6bee640 5 monclient: authenticate success, global_id 4322 2024-08-09T14:50:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.838+0000 7f84d6bee640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.838+0000 7f84d6bee640 4 set_mon_vals no callback set 2024-08-09T14:50:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.838+0000 7f84d6bee640 10 monclient: shutdown 2024-08-09T14:50:52.849 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.838+0000 7f84d6bee640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.838+0000 7f84d6bee640 1 -- 172.21.15.155:0/4258283926 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00b2d80 msgr2=0x7f84d00b4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.838+0000 7f84d6bee640 1 --2- 172.21.15.155:0/4258283926 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00b2d80 0x7f84d00b4df0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f84d00b6520 tx=0x7f84bc02b910 comp rx=0 tx=0).stop 2024-08-09T14:50:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.839+0000 7f84d6bee640 1 -- 172.21.15.155:0/4258283926 shutdown_connections 2024-08-09T14:50:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.839+0000 7f84d6bee640 1 --2- 172.21.15.155:0/4258283926 >> v2:172.21.15.155:3300/0 conn(0x7f84d0142c50 0x7f84d013fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.839+0000 7f84d6bee640 1 --2- 172.21.15.155:0/4258283926 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d00b2d80 0x7f84d00b4df0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.839+0000 7f84d6bee640 1 --2- 172.21.15.155:0/4258283926 >> v2:172.21.15.194:3300/0 conn(0x7f84d006d8d0 0x7f84d00aa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.839+0000 7f84d6bee640 1 -- 172.21.15.155:0/4258283926 >> 172.21.15.155:0/4258283926 conn(0x7f84d00b8d70 msgr2=0x7f84d00b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.839+0000 7f84d6bee640 1 -- 172.21.15.155:0/4258283926 shutdown_connections 2024-08-09T14:50:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.839+0000 7f84d6bee640 1 -- 172.21.15.155:0/4258283926 wait complete. 2024-08-09T14:50:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.840+0000 7f84d6bee640 10 monclient: build_initial_monmap 2024-08-09T14:50:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.840+0000 7f84d6bee640 10 monclient: monmap: 2024-08-09T14:50:52.850 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:52.851 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.840+0000 7f84d6bee640 1 librados: starting msgr at 2024-08-09T14:50:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.840+0000 7f84d6bee640 1 librados: starting objecter 2024-08-09T14:50:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.840+0000 7f84d6bee640 1 Processor -- start 2024-08-09T14:50:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.840+0000 7f84d6bee640 1 -- start start 2024-08-09T14:50:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.840+0000 7f84d6bee640 1 librados: setting wanted keys 2024-08-09T14:50:52.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.840+0000 7f84d6bee640 1 librados: calling monclient init 2024-08-09T14:50:52.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.840+0000 7f84d6bee640 10 monclient: init 2024-08-09T14:50:52.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.841+0000 7f84d6bee640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:52.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.841+0000 7f84d6bee640 10 monclient: _add_conns ranks=[0,1,2] 2024-08-09T14:50:52.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.841+0000 7f84d6bee640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d006d8d0 0x7f84d013eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.841+0000 7f84d6bee640 10 monclient(hunting): picked mon.noname-a con 0x7f84d006d8d0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:52.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.841+0000 7f84d6bee640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84d00b2d80 0x7f84d01cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f84d6bee640 10 monclient(hunting): picked mon.noname-b con 0x7f84d00b2d80 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:52.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f84d6bee640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d0142c50 0x7f84d01cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f84d6bee640 10 monclient(hunting): picked mon.noname-c con 0x7f84d0142c50 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:52.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f84d6bee640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f84d6bee640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f84d0140740 con 0x7f84d006d8d0 2024-08-09T14:50:52.852 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f84d6bee640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.853 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f84d6bee640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f84d01433f0 con 0x7f84d00b2d80 2024-08-09T14:50:52.854 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f8194ff9640 1 -- 172.21.15.155:0/2390153102 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8190044020 con 0x7f81980b26e0 2024-08-09T14:50:52.854 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f8194ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.855 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f8194ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:52.855 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f8194ff9640 10 monclient: dump: 2024-08-09T14:50:52.855 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.855 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.855 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.855 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.855 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.855 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.855 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.855 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.855 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.855 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.855 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f8194ff9640 10 monclient: _finish_auth 0 2024-08-09T14:50:52.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f8194ff9640 10 monclient: _check_auth_tickets 2024-08-09T14:50:52.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f8194ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:52.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f8194ff9640 1 -- 172.21.15.155:0/2390153102 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8190040670 con 0x7f81980b26e0 2024-08-09T14:50:52.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f819d913640 5 monclient: authenticate success, global_id 4325 2024-08-09T14:50:52.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f819d913640 10 monclient: _renew_subs 2024-08-09T14:50:52.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f819d913640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:52.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f819d913640 1 -- 172.21.15.155:0/2390153102 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81980b7700 con 0x7f81980b26e0 2024-08-09T14:50:52.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f8194ff9640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:52.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f8194ff9640 1 -- 172.21.15.155:0/2390153102 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8190040a60 con 0x7f81980b26e0 2024-08-09T14:50:52.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f8194ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f8194ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:52.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f8195ffb640 4 set_mon_vals no callback set 2024-08-09T14:50:52.856 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f8194ff9640 10 monclient: dump: 2024-08-09T14:50:52.856 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.857 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.857 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.857 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.857 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.857 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.857 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.857 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.857 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.857 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f819d913640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:52.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f819d913640 10 monclient: _renew_subs 2024-08-09T14:50:52.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f819d913640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:52.857 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f819d913640 1 -- 172.21.15.155:0/2390153102 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8198144fa0 con 0x7f81980b26e0 2024-08-09T14:50:52.858 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f819d913640 1 librados: init done 2024-08-09T14:50:52.858 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-08-09T14:50:52.858 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-08-09T14:50:52.859 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f84d6bee640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.859 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f84d6bee640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f84d0146ec0 con 0x7f84d0142c50 2024-08-09T14:50:52.859 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f84d6bee640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:52.859 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.842+0000 7f84d6bee640 10 monclient(hunting): authenticate will time out at 1259.879164s 2024-08-09T14:50:52.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.843+0000 7f819d913640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:52.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.843+0000 7f819d913640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:52.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.843+0000 7f819d913640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:52.860 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.843+0000 7f819d913640 1 -- 172.21.15.155:0/2390153102 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8198140510 con 0x7f81980b26e0 2024-08-09T14:50:52.861 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.844+0000 7f3fb1502640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f3fac0b0540 0x7f3fac1469d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.155:40250/0 (socket says 172.21.15.155:40250) 2024-08-09T14:50:52.861 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.844+0000 7f3fb1502640 1 -- 172.21.15.155:0/1594534918 learned_addr learned my addr 172.21.15.155:0/1594534918 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:52.861 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.844+0000 7f3fb2504640 1 --2- 172.21.15.155:0/1594534918 >> v2:172.21.15.194:3300/0 conn(0x7f3fac0b3840 0x7f3fac0b3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.861 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.845+0000 7f3fb2504640 10 monclient(hunting): get_auth_request con 0x7f3fac0b3840 auth_method 0 2024-08-09T14:50:52.861 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.845+0000 7f3fb2504640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:52.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.845+0000 7f3fb2504640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.845+0000 7f3fb2504640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.846+0000 7f3fb2504640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:52.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.846+0000 7f3fb2504640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:52.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.846+0000 7f3fb2504640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:52.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.848+0000 7f3fb2504640 10 monclient(hunting): handle_auth_done global_id 4345 payload 274 2024-08-09T14:50:52.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.848+0000 7f3fb2504640 1 -- 172.21.15.155:0/1594534918 >> v1:172.21.15.194:6789/0 conn(0x7f3fac0b0540 legacy=0x7f3fac1469d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.848+0000 7f3fb2504640 1 -- 172.21.15.155:0/1594534918 >> v1:172.21.15.155:6789/0 conn(0x7f3fac0b0160 legacy=0x7f3fac0b22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:52.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.848+0000 7f3fb2504640 10 monclient: _finish_hunting 0 2024-08-09T14:50:52.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.848+0000 7f3fb2504640 1 monclient: found mon.noname-c 2024-08-09T14:50:52.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.848+0000 7f3fb2504640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:52.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.848+0000 7f3fb2504640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:50:52.862 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.848+0000 7f3fb2504640 1 -- 172.21.15.155:0/1594534918 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fac150580 con 0x7f3fac0b3840 2024-08-09T14:50:52.863 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.855+0000 7f84cffff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84d00b2d80 0x7f84d01cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.863 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.856+0000 7f3fb2504640 1 --2- 172.21.15.155:0/1594534918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fac0b3840 0x7f3fac0b3cd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4009d90 tx=0x7f3fa402bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eefc3388a349863b server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:52.865 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.856+0000 7f84cffff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84d00b2d80 0x7f84d01cc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:48050/0 (socket says 172.21.15.155:48050) 2024-08-09T14:50:52.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.856+0000 7f84cffff640 1 -- 172.21.15.155:0/1379318804 learned_addr learned my addr 172.21.15.155:0/1379318804 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:52.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.856+0000 7f84cffff640 10 monclient(hunting): get_auth_request con 0x7f84d00b2d80 auth_method 0 2024-08-09T14:50:52.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.856+0000 7f84cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:52.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.856+0000 7f84cffff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.856+0000 7f84cffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.857+0000 7f84cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:52.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.857+0000 7f84cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:52.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.857+0000 7f84cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:52.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.860+0000 7f84cffff640 10 monclient(hunting): handle_auth_done global_id 4348 payload 824 2024-08-09T14:50:52.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.860+0000 7f84cffff640 1 -- 172.21.15.155:0/1379318804 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d0142c50 msgr2=0x7f84d01cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.860+0000 7f84cffff640 1 --2- 172.21.15.155:0/1379318804 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d0142c50 0x7f84d01cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.866 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.860+0000 7f84cffff640 1 -- 172.21.15.155:0/1379318804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d006d8d0 msgr2=0x7f84d013eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.860+0000 7f84cffff640 1 --2- 172.21.15.155:0/1379318804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d006d8d0 0x7f84d013eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.860+0000 7f84cffff640 10 monclient: _finish_hunting 0 2024-08-09T14:50:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.860+0000 7f84cffff640 1 monclient: found mon.noname-b 2024-08-09T14:50:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.860+0000 7f84cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.860+0000 7f84cffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.860+0000 7f84cffff640 1 -- 172.21.15.155:0/1379318804 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84d0140a50 con 0x7f84d00b2d80 2024-08-09T14:50:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.863+0000 7f84cffff640 1 --2- 172.21.15.155:0/1379318804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84d00b2d80 0x7f84d01cc1c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f84bc02b950 tx=0x7f84bc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.863+0000 7f84cd7fa640 10 client.?.objecter ms_handle_connect 0x7f84d00b2d80 2024-08-09T14:50:52.867 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.863+0000 7f84cd7fa640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:52.868 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.864+0000 7f3fb0d01640 1 -- 172.21.15.155:0/1594534918 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4039070 con 0x7f3fac0b3840 2024-08-09T14:50:52.868 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.864+0000 7f3fb0d01640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.868 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.864+0000 7f3fb0d01640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:52.868 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.864+0000 7f3fb0d01640 10 monclient: dump: 2024-08-09T14:50:52.868 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.868 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.869 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.864+0000 7f3fb0d01640 10 monclient: _finish_auth 0 2024-08-09T14:50:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.864+0000 7f3fb0d01640 10 monclient: _check_auth_tickets 2024-08-09T14:50:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.864+0000 7f3fb0d01640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.864+0000 7f3fb0d01640 1 -- 172.21.15.155:0/1594534918 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fa402cb40 con 0x7f3fac0b3840 2024-08-09T14:50:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.864+0000 7f3fb0d01640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:52.869 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.864+0000 7f3fb0d01640 1 -- 172.21.15.155:0/1594534918 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4034440 con 0x7f3fac0b3840 2024-08-09T14:50:52.870 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.864+0000 7f3fb0d01640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.870 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.864+0000 7f3fb0d01640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:52.870 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.864+0000 7f3fb0d01640 10 monclient: dump: 2024-08-09T14:50:52.870 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.870 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.870 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.870 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.870 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.870 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.870 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.870 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.870 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.870 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.871 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.865+0000 7f3fb3f67640 5 monclient: authenticate success, global_id 4345 2024-08-09T14:50:52.871 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.865+0000 7f3fb3f67640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:52.871 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.865+0000 7f3fb3f67640 4 set_mon_vals no callback set 2024-08-09T14:50:52.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.865+0000 7f3fb3f67640 10 monclient: shutdown 2024-08-09T14:50:52.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.866+0000 7f3fb3f67640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:52.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.866+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1594534918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fac0b3840 msgr2=0x7f3fac0b3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:52.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.866+0000 7f3fb3f67640 1 --2- 172.21.15.155:0/1594534918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fac0b3840 0x7f3fac0b3cd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f3fa4009d90 tx=0x7f3fa402bd90 comp rx=0 tx=0).stop 2024-08-09T14:50:52.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.866+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1594534918 shutdown_connections 2024-08-09T14:50:52.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.866+0000 7f3fb3f67640 1 --2- 172.21.15.155:0/1594534918 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fac0b3840 0x7f3fac0b3cd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:52.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.866+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1594534918 >> 172.21.15.155:0/1594534918 conn(0x7f3fac06dbd0 msgr2=0x7f3fac0676f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:52.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.866+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1594534918 shutdown_connections 2024-08-09T14:50:52.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.866+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1594534918 wait complete. 2024-08-09T14:50:52.874 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.867+0000 7f3fb3f67640 10 monclient: build_initial_monmap 2024-08-09T14:50:52.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.867+0000 7f3fb3f67640 10 monclient: monmap: 2024-08-09T14:50:52.875 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:52.875 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:52.875 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:52.875 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:52.875 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:52.875 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.875 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:52.875 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:52.875 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:52.875 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.867+0000 7f3fb3f67640 1 librados: starting msgr at 2024-08-09T14:50:52.875 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.867+0000 7f3fb3f67640 1 librados: starting objecter 2024-08-09T14:50:52.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.867+0000 7f3fb3f67640 1 Processor -- start 2024-08-09T14:50:52.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.867+0000 7f3fb3f67640 1 -- start start 2024-08-09T14:50:52.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.867+0000 7f3fb3f67640 1 librados: setting wanted keys 2024-08-09T14:50:52.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.867+0000 7f3fb3f67640 1 librados: calling monclient init 2024-08-09T14:50:52.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.867+0000 7f3fb3f67640 10 monclient: init 2024-08-09T14:50:52.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:52.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 10 monclient: _add_conns ranks=[1,0,2] 2024-08-09T14:50:52.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fac0b0160 0x7f3fac0ae420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 10 monclient(hunting): picked mon.noname-b con 0x7f3fac0b0160 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:52.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fac0b0540 0x7f3fac0ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 10 monclient(hunting): picked mon.noname-a con 0x7f3fac0b0540 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:52.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fac0b3840 0x7f3fac1d5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 10 monclient(hunting): picked mon.noname-c con 0x7f3fac0b3840 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:52.876 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f3fac14e930 con 0x7f3fac0b0540 2024-08-09T14:50:52.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3fac14e7b0 con 0x7f3fac0b0160 2024-08-09T14:50:52.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f3fac14eab0 con 0x7f3fac0b3840 2024-08-09T14:50:52.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:52.877 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.868+0000 7f3fb3f67640 10 monclient(hunting): authenticate will time out at 1259.905956s 2024-08-09T14:50:52.879 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84cd7fa640 1 -- 172.21.15.155:0/1379318804 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc0443f0 con 0x7f84d00b2d80 2024-08-09T14:50:52.879 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.879 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84cd7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:52.879 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84cd7fa640 10 monclient: dump: 2024-08-09T14:50:52.880 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.880 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.880 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.880 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.880 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.880 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.880 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.880 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.880 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.880 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.880 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84cd7fa640 10 monclient: _finish_auth 0 2024-08-09T14:50:52.880 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84cd7fa640 10 monclient: _check_auth_tickets 2024-08-09T14:50:52.880 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:52.880 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84cd7fa640 1 -- 172.21.15.155:0/1379318804 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f84bc044590 con 0x7f84d00b2d80 2024-08-09T14:50:52.881 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84cd7fa640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:52.881 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84cd7fa640 1 -- 172.21.15.155:0/1379318804 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84bc044980 con 0x7f84d00b2d80 2024-08-09T14:50:52.881 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:52.881 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:52.881 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84cd7fa640 10 monclient: dump: 2024-08-09T14:50:52.881 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:52.881 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.881 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.881 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:52.881 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:52.881 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.881 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:52.881 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:52.882 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:52.882 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.882 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84d6bee640 5 monclient: authenticate success, global_id 4348 2024-08-09T14:50:52.882 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84d6bee640 10 monclient: _renew_subs 2024-08-09T14:50:52.882 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84ceffd640 4 set_mon_vals no callback set 2024-08-09T14:50:52.882 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84d6bee640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:52.882 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84d6bee640 1 -- 172.21.15.155:0/1379318804 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84d01cdd30 con 0x7f84d00b2d80 2024-08-09T14:50:52.882 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84d6bee640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:52.882 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84d6bee640 10 monclient: _renew_subs 2024-08-09T14:50:52.882 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84d6bee640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:52.882 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84d6bee640 1 -- 172.21.15.155:0/1379318804 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84d01ce2f0 con 0x7f84d00b2d80 2024-08-09T14:50:52.882 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.870+0000 7f84d6bee640 1 librados: init done 2024-08-09T14:50:52.883 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.871+0000 7f3fb1d03640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fac0b0160 0x7f3fac0ae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.883 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.871+0000 7f84d6bee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:52.883 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.872+0000 7f84d6bee640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:52.883 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.872+0000 7f84d6bee640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:52.883 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.872+0000 7f84d6bee640 1 -- 172.21.15.155:0/1379318804 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84d00b4df0 con 0x7f84d00b2d80 2024-08-09T14:50:52.884 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.883+0000 7f3fb1d03640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fac0b0160 0x7f3fac0ae420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:48058/0 (socket says 172.21.15.155:48058) 2024-08-09T14:50:52.884 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.883+0000 7f3fb1d03640 1 -- 172.21.15.155:0/1916473263 learned_addr learned my addr 172.21.15.155:0/1916473263 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:52.884 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.883+0000 7f3fb1d03640 10 monclient(hunting): get_auth_request con 0x7f3fac0b0160 auth_method 0 2024-08-09T14:50:52.884 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.883+0000 7f3fb1d03640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:52.884 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.883+0000 7f3fb1d03640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.884 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.883+0000 7f3fb1d03640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.891+0000 7f84cd7fa640 1 -- 172.21.15.155:0/1379318804 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f84bc038020 con 0x7f84d00b2d80 2024-08-09T14:50:52.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.892+0000 7f84cd7fa640 1 --2- 172.21.15.155:0/1379318804 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f84b406d810 0x7f84b406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.893 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.892+0000 7f84d498a640 1 --2- 172.21.15.155:0/1379318804 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f84b406d810 0x7f84b406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.892+0000 7f84d498a640 10 monclient: get_auth_request con 0x7f84b406d810 auth_method 0 2024-08-09T14:50:52.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.893+0000 7f84cd7fa640 10 client.4348.objecter ms_handle_connect 0x7f84b406d810 2024-08-09T14:50:52.894 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.893+0000 7f84d498a640 1 --2- 172.21.15.155:0/1379318804 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f84b406d810 0x7f84b406fcd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009710 tx=0x7f84c4006c90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:52.921 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-08-09T14:50:52.968 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.967+0000 7f3fb2504640 1 --2- 172.21.15.155:0/1916473263 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fac0b3840 0x7f3fac1d5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.967+0000 7f3fb2504640 10 monclient(hunting): get_auth_request con 0x7f3fac0b3840 auth_method 0 2024-08-09T14:50:52.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.967+0000 7f3fb2504640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:52.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.967+0000 7f3fb2504640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:52.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.967+0000 7f3fb2504640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:52.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.968+0000 7f3fb2504640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:52.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.968+0000 7f3fb2504640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:52.969 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.968+0000 7f3fb2504640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:52.981 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.979+0000 7f8194ff9640 1 -- 172.21.15.155:0/2390153102 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8190047020 con 0x7f81980b26e0 2024-08-09T14:50:52.981 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.980+0000 7f8194ff9640 1 --2- 172.21.15.155:0/2390153102 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f817806d810 0x7f817806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.981 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.980+0000 7f8196ffd640 1 --2- 172.21.15.155:0/2390153102 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f817806d810 0x7f817806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:52.981 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.980+0000 7f8196ffd640 10 monclient: get_auth_request con 0x7f817806d810 auth_method 0 2024-08-09T14:50:52.982 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.981+0000 7f8196ffd640 1 --2- 172.21.15.155:0/2390153102 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f817806d810 0x7f817806fcd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8188009670 tx=0x7f8188006bf0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:52.982 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.981+0000 7f8194ff9640 10 client.4325.objecter ms_handle_connect 0x7f817806d810 2024-08-09T14:50:52.984 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.983+0000 7f8912a20640 10 monclient: get_monmap_and_config 2024-08-09T14:50:52.984 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.983+0000 7f8912a20640 10 monclient: build_initial_monmap 2024-08-09T14:50:52.984 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.983+0000 7f8912a20640 10 monclient: monmap: 2024-08-09T14:50:52.984 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:52.984 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:52.984 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:52.984259+0000 2024-08-09T14:50:52.985 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:52.984259+0000 2024-08-09T14:50:52.985 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:52.985 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:52.985 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:52.985 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:52.985 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:52.985 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:52.985 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:52.985 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:52.987 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.984+0000 7f8912a20640 1 Processor -- start 2024-08-09T14:50:52.987 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.984+0000 7f8912a20640 1 -- start start 2024-08-09T14:50:52.987 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.984+0000 7f8912a20640 10 monclient: init 2024-08-09T14:50:52.987 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.985+0000 7f8912a20640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:52.987 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.985+0000 7f8912a20640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-08-09T14:50:52.987 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.985+0000 7f8912a20640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f890c06d8d0 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:52.987 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.985+0000 7f8912a20640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f890c0b2d80 0x7f890c0aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.987 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.985+0000 7f8912a20640 10 monclient(hunting): picked mon.noname-b con 0x7f890c0b2d80 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:52.987 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.985+0000 7f8912a20640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f890c0b4df0 0x7f890c13fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:52.987 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.985+0000 7f8912a20640 10 monclient(hunting): picked mon.noname-c con 0x7f890c0b4df0 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:52.988 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.985+0000 7f8912a20640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f890c143360 con 0x7f890c06d8d0 2024-08-09T14:50:52.988 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.985+0000 7f8912a20640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.988 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.985+0000 7f8912a20640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f890c06dcb0 con 0x7f890c0b4df0 2024-08-09T14:50:52.988 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.985+0000 7f8912a20640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:52.988 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.985+0000 7f8912a20640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f890c0b4680 con 0x7f890c0b2d80 2024-08-09T14:50:52.988 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.985+0000 7f8912a20640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:52.988 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.985+0000 7f8912a20640 10 monclient(hunting): authenticate will time out at 1260.022318s 2024-08-09T14:50:52.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.996+0000 7f8194ff9640 1 -- 172.21.15.155:0/2390153102 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f819007a670 con 0x7f81980b26e0 2024-08-09T14:50:52.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.996+0000 7f8194ff9640 10 client.4325.objecter ms_dispatch 0x7f8198000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:52.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.997+0000 7f8194ff9640 3 client.4325.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:52.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.997+0000 7f8194ff9640 3 client.4325.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:52.998 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:52.997+0000 7f8194ff9640 20 client.4325.objecter dump_active .. 0 homeless 2024-08-09T14:50:53.002 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.001+0000 7f3fb2504640 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2024-08-09T14:50:53.002 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.001+0000 7f3fb2504640 1 -- 172.21.15.155:0/1916473263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fac0b0160 msgr2=0x7f3fac0ae420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.002 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.001+0000 7f3fb2504640 1 --2- 172.21.15.155:0/1916473263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fac0b0160 0x7f3fac0ae420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.002 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.001+0000 7f3fb2504640 1 -- 172.21.15.155:0/1916473263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fac0b0540 msgr2=0x7f3fac0ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.003 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.001+0000 7f3fb2504640 1 --2- 172.21.15.155:0/1916473263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fac0b0540 0x7f3fac0ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.003 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.001+0000 7f3fb2504640 10 monclient: _finish_hunting 0 2024-08-09T14:50:53.003 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.001+0000 7f3fb2504640 1 monclient: found mon.noname-c 2024-08-09T14:50:53.003 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.001+0000 7f3fb2504640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:53.003 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.001+0000 7f3fb2504640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:50:53.003 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.001+0000 7f3fb2504640 1 -- 172.21.15.155:0/1916473263 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fac0aa9c0 con 0x7f3fac0b3840 2024-08-09T14:50:53.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.021+0000 7f8fda9d4640 10 monclient: get_monmap_and_config 2024-08-09T14:50:53.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.021+0000 7f8fda9d4640 10 monclient: build_initial_monmap 2024-08-09T14:50:53.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.021+0000 7f8fda9d4640 10 monclient: monmap: 2024-08-09T14:50:53.023 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:53.023 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.023 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:53.022970+0000 2024-08-09T14:50:53.023 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:53.022970+0000 2024-08-09T14:50:53.023 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:53.023 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.024 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:53.024 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:53.024 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:53.024 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:53.024 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:53.024 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.022+0000 7f8fda9d4640 1 Processor -- start 2024-08-09T14:50:53.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.022+0000 7f8fda9d4640 1 -- start start 2024-08-09T14:50:53.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.022+0000 7f8fda9d4640 10 monclient: init 2024-08-09T14:50:53.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.023+0000 7f8fda9d4640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:53.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.023+0000 7f8fda9d4640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-08-09T14:50:53.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.023+0000 7f8fda9d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8fd406d8d0 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:53.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.023+0000 7f8fda9d4640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fd40b2d80 0x7f8fd40aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.023+0000 7f8fda9d4640 10 monclient(hunting): picked mon.noname-b con 0x7f8fd40b2d80 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:53.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.023+0000 7f8fda9d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8fd40b4df0 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:53.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.023+0000 7f8fda9d4640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fd4143360 con 0x7f8fd406d8d0 2024-08-09T14:50:53.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.023+0000 7f8fda9d4640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fd4140700 con 0x7f8fd40b4df0 2024-08-09T14:50:53.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.023+0000 7f8fda9d4640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:53.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.023+0000 7f8fda9d4640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f8fd406dcb0 con 0x7f8fd40b2d80 2024-08-09T14:50:53.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.023+0000 7f8fda9d4640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:53.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.023+0000 7f8fda9d4640 10 monclient(hunting): authenticate will time out at 1260.060612s 2024-08-09T14:50:53.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.056+0000 7f8bae5fa640 10 monclient: get_monmap_and_config 2024-08-09T14:50:53.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.056+0000 7f8bae5fa640 10 monclient: build_initial_monmap 2024-08-09T14:50:53.057 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.056+0000 7f8bae5fa640 10 monclient: monmap: 2024-08-09T14:50:53.057 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:53.058 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.058 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:53.057300+0000 2024-08-09T14:50:53.058 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:53.057300+0000 2024-08-09T14:50:53.058 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:53.058 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.058 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:53.058 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:53.058 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:53.058 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:53.058 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:53.058 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.056+0000 7f8bae5fa640 1 Processor -- start 2024-08-09T14:50:53.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 1 -- start start 2024-08-09T14:50:53.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 10 monclient: init 2024-08-09T14:50:53.059 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f4d42b2e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-08-09T14:50:53.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f4d42b2e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-08-09T14:50:53.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f4d42b2e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:53.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f4d42b2e640 1 -- 172.21.15.155:0/3868600138 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f4d3c068960 con 0x7f4d3c0b0540 2024-08-09T14:50:53.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:53.060 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-08-09T14:50:53.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f8ba806d8d0 0x7f8ba80aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba806d8d0 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:53.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ba80b2d80 0x7f8ba80b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 10 monclient(hunting): picked mon.noname-b con 0x7f8ba80b2d80 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:53.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ba8142c50 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:53.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ba8140700 con 0x7f8ba8142c50 2024-08-09T14:50:53.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:53.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f8ba806dcb0 con 0x7f8ba806d8d0 2024-08-09T14:50:53.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:53.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba81408d0 con 0x7f8ba80b2d80 2024-08-09T14:50:53.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:53.061 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.057+0000 7f8bae5fa640 10 monclient(hunting): authenticate will time out at 1260.094740s 2024-08-09T14:50:53.124 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.123+0000 7f84cd7fa640 1 -- 172.21.15.155:0/1379318804 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f84bc079e30 con 0x7f84d00b2d80 2024-08-09T14:50:53.124 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.123+0000 7f84cd7fa640 10 client.4348.objecter ms_dispatch 0x7f84d0000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:53.124 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.123+0000 7f84cd7fa640 3 client.4348.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:53.124 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.123+0000 7f84cd7fa640 3 client.4348.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:53.124 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.123+0000 7f84cd7fa640 20 client.4348.objecter dump_active .. 0 homeless 2024-08-09T14:50:53.126 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.125+0000 7f890bfff640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7f890c06d8d0 0x7f890c142c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:34580/0 (socket says 172.21.15.155:34580) 2024-08-09T14:50:53.126 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.125+0000 7f890bfff640 1 -- 172.21.15.155:0/140100764 learned_addr learned my addr 172.21.15.155:0/140100764 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:53.135 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.133+0000 7f84cd7fa640 1 -- 172.21.15.155:0/1379318804 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f84bc03f070 con 0x7f84d00b2d80 2024-08-09T14:50:53.135 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.134+0000 7f84cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.135 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.134+0000 7f84cd7fa640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:53.151 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.150+0000 7f8ba7fff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f8ba806d8d0 0x7f8ba80aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:53.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.151+0000 7f8ba7fff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f8ba806d8d0 0x7f8ba80aa9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:48070/0 (socket says 172.21.15.155:48070) 2024-08-09T14:50:53.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.151+0000 7f8ba7fff640 1 -- 172.21.15.155:0/3731816366 learned_addr learned my addr 172.21.15.155:0/3731816366 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:53.152 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.151+0000 7f8ba7fff640 10 monclient(hunting): get_auth_request con 0x7f8ba806d8d0 auth_method 0 2024-08-09T14:50:53.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.151+0000 7f8ba7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:53.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.151+0000 7f8ba7fff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:53.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.151+0000 7f8ba7fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:53.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.151+0000 7f8fd8f71640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7f8fd40b4df0 0x7f8fd413fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.155:40272/0 (socket says 172.21.15.155:40272) 2024-08-09T14:50:53.153 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.151+0000 7f8fd8f71640 1 -- 172.21.15.155:0/1689502221 learned_addr learned my addr 172.21.15.155:0/1689502221 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:53.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.152+0000 7f8ba7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:53.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.152+0000 7f8ba7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:53.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.152+0000 7f8ba7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:53.154 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.153+0000 7f8910fbd640 1 --2- 172.21.15.155:0/140100764 >> v2:172.21.15.194:3300/0 conn(0x7f890c0b4df0 0x7f890c13fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:53.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.154+0000 7f8ba7fff640 10 monclient(hunting): handle_auth_done global_id 4354 payload 274 2024-08-09T14:50:53.155 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.154+0000 7f8ba7fff640 1 -- 172.21.15.155:0/3731816366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ba80b2d80 msgr2=0x7f8ba80b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.154+0000 7f8ba7fff640 1 --2- 172.21.15.155:0/3731816366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ba80b2d80 0x7f8ba80b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.154+0000 7f8ba7fff640 1 -- 172.21.15.155:0/3731816366 >> v1:172.21.15.194:6789/0 conn(0x7f8ba8142c50 legacy=0x7f8ba813fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.154+0000 7f8ba7fff640 10 monclient: _finish_hunting 0 2024-08-09T14:50:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.154+0000 7f8ba7fff640 1 monclient: found mon.noname-c 2024-08-09T14:50:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.154+0000 7f8ba7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.154+0000 7f8ba7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:50:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.154+0000 7f8ba7fff640 1 -- 172.21.15.155:0/3731816366 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba8143100 con 0x7f8ba806d8d0 2024-08-09T14:50:53.156 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.155+0000 7f8910fbd640 10 monclient(hunting): get_auth_request con 0x7f890c0b4df0 auth_method 0 2024-08-09T14:50:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.155+0000 7f8910fbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.155+0000 7f8910fbd640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.155+0000 7f8910fbd640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.156+0000 7f8910fbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.156+0000 7f8910fbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:53.157 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.156+0000 7f8910fbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:53.158 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.157+0000 7f8910fbd640 10 monclient(hunting): handle_auth_done global_id 4357 payload 274 2024-08-09T14:50:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.157+0000 7f8910fbd640 1 -- 172.21.15.155:0/140100764 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f890c0b2d80 msgr2=0x7f890c0aa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.157+0000 7f8910fbd640 1 --2- 172.21.15.155:0/140100764 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f890c0b2d80 0x7f890c0aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.157+0000 7f8910fbd640 1 -- 172.21.15.155:0/140100764 >> v1:172.21.15.155:6789/0 conn(0x7f890c06d8d0 legacy=0x7f890c142c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.157+0000 7f8910fbd640 10 monclient: _finish_hunting 0 2024-08-09T14:50:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.157+0000 7f8910fbd640 1 monclient: found mon.noname-c 2024-08-09T14:50:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.157+0000 7f8910fbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.157+0000 7f8910fbd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:50:53.159 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.157+0000 7f8910fbd640 1 -- 172.21.15.155:0/140100764 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f890c140530 con 0x7f890c0b4df0 2024-08-09T14:50:53.162 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.161+0000 7f8ba7fff640 1 --2- 172.21.15.155:0/3731816366 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ba806d8d0 0x7f8ba80aa9d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009870 tx=0x7f8b9802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97c807b53bc68233 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:53.163 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.162+0000 7f8910fbd640 1 --2- 172.21.15.155:0/140100764 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f890c0b4df0 0x7f890c13fff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8904009d90 tx=0x7f890402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f98a2c8bd6d51c2 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:53.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f890affd640 1 -- 172.21.15.155:0/140100764 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904039070 con 0x7f890c0b4df0 2024-08-09T14:50:53.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f890affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:53.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f890affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:53.168 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f890affd640 10 monclient: dump: 2024-08-09T14:50:53.168 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:53.168 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.168 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.168 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.168 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:53.168 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.168 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:53.169 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:53.169 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:53.169 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f890affd640 10 monclient: _finish_auth 0 2024-08-09T14:50:53.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f890affd640 10 monclient: _check_auth_tickets 2024-08-09T14:50:53.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f890affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:53.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f890affd640 1 -- 172.21.15.155:0/140100764 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f890402cc50 con 0x7f890c0b4df0 2024-08-09T14:50:53.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f8912a20640 5 monclient: authenticate success, global_id 4357 2024-08-09T14:50:53.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f8912a20640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:53.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f890affd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:53.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f8912a20640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:53.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f890affd640 1 -- 172.21.15.155:0/140100764 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904034440 con 0x7f890c0b4df0 2024-08-09T14:50:53.169 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f8912a20640 4 set_mon_vals no callback set 2024-08-09T14:50:53.170 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f890affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:53.170 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f890affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:53.170 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.166+0000 7f890affd640 10 monclient: dump: 2024-08-09T14:50:53.170 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:53.170 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.170 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.170 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.170 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:53.170 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.170 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:53.170 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:53.170 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:53.170 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.167+0000 7f8912a20640 10 monclient: shutdown 2024-08-09T14:50:53.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.167+0000 7f8912a20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:53.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.167+0000 7f8912a20640 1 -- 172.21.15.155:0/140100764 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f890c0b4df0 msgr2=0x7f890c13fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.167+0000 7f8912a20640 1 --2- 172.21.15.155:0/140100764 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f890c0b4df0 0x7f890c13fff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8904009d90 tx=0x7f890402bea0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.167+0000 7f8912a20640 1 -- 172.21.15.155:0/140100764 shutdown_connections 2024-08-09T14:50:53.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.167+0000 7f8912a20640 1 --2- 172.21.15.155:0/140100764 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f890c0b4df0 0x7f890c13fff0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.167+0000 7f8912a20640 1 --2- 172.21.15.155:0/140100764 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f890c0b2d80 0x7f890c0aa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.167+0000 7f8912a20640 1 -- 172.21.15.155:0/140100764 >> 172.21.15.155:0/140100764 conn(0x7f890c0b8d70 msgr2=0x7f890c0b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:53.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.167+0000 7f8912a20640 1 -- 172.21.15.155:0/140100764 shutdown_connections 2024-08-09T14:50:53.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.167+0000 7f8912a20640 1 -- 172.21.15.155:0/140100764 wait complete. 2024-08-09T14:50:53.173 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.168+0000 7f8912a20640 10 monclient: build_initial_monmap 2024-08-09T14:50:53.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.168+0000 7f8912a20640 10 monclient: monmap: 2024-08-09T14:50:53.174 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:53.174 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:53.174 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:53.174 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:53.174 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:53.174 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.174 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:53.174 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:53.174 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:53.174 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.174 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.168+0000 7f8912a20640 1 librados: starting msgr at 2024-08-09T14:50:53.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.168+0000 7f8912a20640 1 librados: starting objecter 2024-08-09T14:50:53.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.168+0000 7f8912a20640 1 Processor -- start 2024-08-09T14:50:53.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.168+0000 7f8912a20640 1 -- start start 2024-08-09T14:50:53.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.168+0000 7f8912a20640 1 librados: setting wanted keys 2024-08-09T14:50:53.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.168+0000 7f8912a20640 1 librados: calling monclient init 2024-08-09T14:50:53.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.168+0000 7f8912a20640 10 monclient: init 2024-08-09T14:50:53.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:53.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 10 monclient: _add_conns ranks=[0,2,1] 2024-08-09T14:50:53.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f890c06d8d0 0x7f890c149ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 10 monclient(hunting): picked mon.noname-a con 0x7f890c06d8d0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:53.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f890c0b2d80 0x7f890c147280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 10 monclient(hunting): picked mon.noname-c con 0x7f890c0b2d80 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:53.175 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f890c0b4df0 0x7f890c1477c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 10 monclient(hunting): picked mon.noname-b con 0x7f890c0b4df0 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:53.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:53.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f890c0b4960 con 0x7f890c06d8d0 2024-08-09T14:50:53.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:53.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f890c1408d0 con 0x7f890c0b4df0 2024-08-09T14:50:53.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:53.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f890c14d6f0 con 0x7f890c0b2d80 2024-08-09T14:50:53.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:53.176 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.169+0000 7f8912a20640 10 monclient(hunting): authenticate will time out at 1260.206490s 2024-08-09T14:50:53.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8ba6ffd640 1 -- 172.21.15.155:0/3731816366 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b98035020 con 0x7f8ba806d8d0 2024-08-09T14:50:53.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8ba6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:53.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8ba6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:53.177 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8ba6ffd640 10 monclient: dump: 2024-08-09T14:50:53.177 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:53.177 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.177 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.177 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.177 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:53.177 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.177 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:53.178 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:53.178 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:53.178 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8ba6ffd640 10 monclient: _finish_auth 0 2024-08-09T14:50:53.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8ba6ffd640 10 monclient: _check_auth_tickets 2024-08-09T14:50:53.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8ba6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:53.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8ba6ffd640 1 -- 172.21.15.155:0/3731816366 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b98031640 con 0x7f8ba806d8d0 2024-08-09T14:50:53.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8bae5fa640 5 monclient: authenticate success, global_id 4354 2024-08-09T14:50:53.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8ba6ffd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:53.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8bae5fa640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:53.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8bae5fa640 4 set_mon_vals no callback set 2024-08-09T14:50:53.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8ba6ffd640 1 -- 172.21.15.155:0/3731816366 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9802d040 con 0x7f8ba806d8d0 2024-08-09T14:50:53.178 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8ba6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:53.179 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8ba6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:53.179 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f8ba6ffd640 10 monclient: dump: 2024-08-09T14:50:53.179 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:53.179 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.179 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.179 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.179 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:53.179 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.179 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:53.179 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:53.179 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:53.179 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.180 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f4d397fa640 1 -- 172.21.15.155:0/3868600138 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4d30040070 con 0x7f4d3c0b0540 2024-08-09T14:50:53.180 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f4d397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-08-09T14:50:53.180 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.173+0000 7f4d397fa640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:53.180 INFO:teuthology.orchestra.run.smithi155.stdout:60129542151 2024-08-09T14:50:53.181 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.174+0000 7f8bae5fa640 10 monclient: shutdown 2024-08-09T14:50:53.181 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.174+0000 7f8bae5fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:53.181 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.174+0000 7f8bae5fa640 1 -- 172.21.15.155:0/3731816366 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ba806d8d0 msgr2=0x7f8ba80aa9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.181 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.174+0000 7f8bae5fa640 1 --2- 172.21.15.155:0/3731816366 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ba806d8d0 0x7f8ba80aa9d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009870 tx=0x7f8b9802b8f0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.181 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.174+0000 7f8bae5fa640 1 -- 172.21.15.155:0/3731816366 shutdown_connections 2024-08-09T14:50:53.181 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.174+0000 7f8bae5fa640 1 --2- 172.21.15.155:0/3731816366 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ba80b2d80 0x7f8ba80b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.181 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.174+0000 7f8bae5fa640 1 --2- 172.21.15.155:0/3731816366 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ba806d8d0 0x7f8ba80aa9d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.174+0000 7f8bae5fa640 1 -- 172.21.15.155:0/3731816366 >> 172.21.15.155:0/3731816366 conn(0x7f8ba80b8d70 msgr2=0x7f8ba80b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:53.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.174+0000 7f8bae5fa640 1 -- 172.21.15.155:0/3731816366 shutdown_connections 2024-08-09T14:50:53.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.174+0000 7f8bae5fa640 1 -- 172.21.15.155:0/3731816366 wait complete. 2024-08-09T14:50:53.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8910fbd640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f890c0b4df0 0x7f890c1477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:53.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8bae5fa640 10 monclient: build_initial_monmap 2024-08-09T14:50:53.182 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8bae5fa640 10 monclient: monmap: 2024-08-09T14:50:53.183 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:53.183 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:53.183 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:53.183 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:53.183 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:53.183 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.183 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:53.183 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:53.183 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:53.183 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8910fbd640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f890c0b4df0 0x7f890c1477c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:48074/0 (socket says 172.21.15.155:48074) 2024-08-09T14:50:53.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8910fbd640 1 -- 172.21.15.155:0/2885178224 learned_addr learned my addr 172.21.15.155:0/2885178224 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:53.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8910fbd640 10 monclient(hunting): get_auth_request con 0x7f890c0b4df0 auth_method 0 2024-08-09T14:50:53.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8910fbd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:53.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8910fbd640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:53.184 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8910fbd640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:53.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8bae5fa640 1 librados: starting msgr at 2024-08-09T14:50:53.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8bae5fa640 1 librados: starting objecter 2024-08-09T14:50:53.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8bae5fa640 1 Processor -- start 2024-08-09T14:50:53.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8bae5fa640 1 -- start start 2024-08-09T14:50:53.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8bae5fa640 1 librados: setting wanted keys 2024-08-09T14:50:53.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8bae5fa640 1 librados: calling monclient init 2024-08-09T14:50:53.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.175+0000 7f8bae5fa640 10 monclient: init 2024-08-09T14:50:53.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8910fbd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:53.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8910fbd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:53.185 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8910fbd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:53.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:53.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-08-09T14:50:53.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 10 librados: watch_flush enter 2024-08-09T14:50:53.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 10 librados: watch_flush exit 2024-08-09T14:50:53.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 20 client.4342.objecter shutdown clearing up homeless session... 2024-08-09T14:50:53.186 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 10 client.4342.objecter successfully canceled tick 2024-08-09T14:50:53.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ba806d8d0 0x7f8ba814a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba806d8d0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:53.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ba80b2d80 0x7f8ba8147b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 10 monclient(hunting): picked mon.noname-b con 0x7f8ba80b2d80 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:53.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ba8142c50 0x7f8ba81480b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.187 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 10 monclient(hunting): picked mon.noname-a con 0x7f8ba8142c50 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:53.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:53.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba80b4960 con 0x7f8ba8142c50 2024-08-09T14:50:53.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:53.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba80b4680 con 0x7f8ba80b2d80 2024-08-09T14:50:53.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:53.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f8ba814d6f0 con 0x7f8ba806d8d0 2024-08-09T14:50:53.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:53.188 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f8bae5fa640 10 monclient(hunting): authenticate will time out at 1260.213634s 2024-08-09T14:50:53.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 1 -- 172.21.15.155:0/3868600138 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f4d2806d810 msgr2=0x7f4d2806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 1 --2- 172.21.15.155:0/3868600138 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f4d2806d810 0x7f4d2806fcd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c001780 tx=0x7f4d2c009040 comp rx=0 tx=0).stop 2024-08-09T14:50:53.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 10 monclient: shutdown 2024-08-09T14:50:53.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:53.189 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 1 -- 172.21.15.155:0/3868600138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d3c0b0540 msgr2=0x7f4d3c13e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 1 --2- 172.21.15.155:0/3868600138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d3c0b0540 0x7f4d3c13e2f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4d30034040 tx=0x7f4d300046c0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 1 -- 172.21.15.155:0/3868600138 shutdown_connections 2024-08-09T14:50:53.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 1 --2- 172.21.15.155:0/3868600138 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f4d2806d810 0x7f4d2806fcd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 1 --2- 172.21.15.155:0/3868600138 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f4d3c146f10 0x7f4d3c13e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 1 --2- 172.21.15.155:0/3868600138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d3c0b0540 0x7f4d3c13e2f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 1 --2- 172.21.15.155:0/3868600138 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f4d3c0b0160 0x7f4d3c145580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 1 -- 172.21.15.155:0/3868600138 >> 172.21.15.155:0/3868600138 conn(0x7f4d3c06dbd0 msgr2=0x7f4d3c075210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:53.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.176+0000 7f4d42b2e640 1 -- 172.21.15.155:0/3868600138 shutdown_connections 2024-08-09T14:50:53.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.177+0000 7f4d42b2e640 1 -- 172.21.15.155:0/3868600138 wait complete. 2024-08-09T14:50:53.190 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.177+0000 7f4d42b2e640 1 librados: shutdown 2024-08-09T14:50:53.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.179+0000 7f8910fbd640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2024-08-09T14:50:53.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.179+0000 7f8910fbd640 1 -- 172.21.15.155:0/2885178224 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f890c0b2d80 msgr2=0x7f890c147280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.179+0000 7f8910fbd640 1 --2- 172.21.15.155:0/2885178224 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f890c0b2d80 0x7f890c147280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.179+0000 7f8910fbd640 1 -- 172.21.15.155:0/2885178224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f890c06d8d0 msgr2=0x7f890c149ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.179+0000 7f8910fbd640 1 --2- 172.21.15.155:0/2885178224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f890c06d8d0 0x7f890c149ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.179+0000 7f8910fbd640 10 monclient: _finish_hunting 0 2024-08-09T14:50:53.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.179+0000 7f8910fbd640 1 monclient: found mon.noname-b 2024-08-09T14:50:53.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.179+0000 7f8910fbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:53.191 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.179+0000 7f8910fbd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:53.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.179+0000 7f8910fbd640 1 -- 172.21.15.155:0/2885178224 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f890c14bcf0 con 0x7f890c0b4df0 2024-08-09T14:50:53.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.183+0000 7f8ba77fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ba80b2d80 0x7f8ba8147b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:53.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.183+0000 7f8ba77fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ba80b2d80 0x7f8ba8147b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:48084/0 (socket says 172.21.15.155:48084) 2024-08-09T14:50:53.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.183+0000 7f8ba77fe640 1 -- 172.21.15.155:0/628073468 learned_addr learned my addr 172.21.15.155:0/628073468 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:53.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.183+0000 7f8ba77fe640 10 monclient(hunting): get_auth_request con 0x7f8ba80b2d80 auth_method 0 2024-08-09T14:50:53.192 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.183+0000 7f8ba77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:53.193 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.183+0000 7f8ba77fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:53.193 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.183+0000 7f8ba77fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:53.193 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.184+0000 7f8ba77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:53.193 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.184+0000 7f8ba77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:53.193 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.184+0000 7f8ba77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:53.193 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.185+0000 7f8910fbd640 1 --2- 172.21.15.155:0/2885178224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f890c0b4df0 0x7f890c1477c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8904009d90 tx=0x7f8904004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:53.193 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.185+0000 7f8908ff9640 10 client.?.objecter ms_handle_connect 0x7f890c0b4df0 2024-08-09T14:50:53.193 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.185+0000 7f8908ff9640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:53.194 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.187+0000 7f8ba77fe640 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2024-08-09T14:50:53.194 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.187+0000 7f8ba77fe640 1 -- 172.21.15.155:0/628073468 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ba806d8d0 msgr2=0x7f8ba814a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.194 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.187+0000 7f8ba77fe640 1 --2- 172.21.15.155:0/628073468 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ba806d8d0 0x7f8ba814a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.194 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.187+0000 7f8ba77fe640 1 -- 172.21.15.155:0/628073468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ba8142c50 msgr2=0x7f8ba81480b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.194 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.187+0000 7f8ba77fe640 1 --2- 172.21.15.155:0/628073468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ba8142c50 0x7f8ba81480b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.194 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.187+0000 7f8ba77fe640 10 monclient: _finish_hunting 0 2024-08-09T14:50:53.194 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.187+0000 7f8ba77fe640 1 monclient: found mon.noname-b 2024-08-09T14:50:53.194 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.187+0000 7f8ba77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:53.195 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.187+0000 7f8ba77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:53.195 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.187+0000 7f8ba77fe640 1 -- 172.21.15.155:0/628073468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba814a1d0 con 0x7f8ba80b2d80 2024-08-09T14:50:53.195 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.188+0000 7f8194ff9640 1 -- 172.21.15.155:0/2390153102 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f819004e030 con 0x7f81980b26e0 2024-08-09T14:50:53.195 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.188+0000 7f8194ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.195 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.188+0000 7f8194ff9640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:53.195 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.189+0000 7f3fb2504640 1 --2- 172.21.15.155:0/1916473263 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fac0b3840 0x7f3fac1d5980 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3fac068420 tx=0x7f3fa4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:53.195 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.189+0000 7f3fa27fc640 10 client.?.objecter ms_handle_connect 0x7f3fac0b3840 2024-08-09T14:50:53.196 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.189+0000 7f3fa27fc640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:53.196 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.6 2024-08-09T14:50:53.197 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8908ff9640 1 -- 172.21.15.155:0/2885178224 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904044020 con 0x7f890c0b4df0 2024-08-09T14:50:53.197 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8908ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:53.198 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8908ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:53.198 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8908ff9640 10 monclient: dump: 2024-08-09T14:50:53.198 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:53.198 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.198 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.198 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.198 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:53.198 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.198 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:53.198 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:53.198 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:53.198 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.198 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8908ff9640 10 monclient: _finish_auth 0 2024-08-09T14:50:53.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8908ff9640 10 monclient: _check_auth_tickets 2024-08-09T14:50:53.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8908ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:53.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8908ff9640 1 -- 172.21.15.155:0/2885178224 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f890402ca50 con 0x7f890c0b4df0 2024-08-09T14:50:53.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8912a20640 5 monclient: authenticate success, global_id 4360 2024-08-09T14:50:53.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8908ff9640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:53.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8908ff9640 1 -- 172.21.15.155:0/2885178224 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8904039040 con 0x7f890c0b4df0 2024-08-09T14:50:53.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8908ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:53.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8908ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:53.199 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8908ff9640 10 monclient: dump: 2024-08-09T14:50:53.199 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:53.199 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.199 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.199 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.200 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:53.200 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.200 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:53.200 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:53.200 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:53.200 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.200 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8912a20640 10 monclient: _renew_subs 2024-08-09T14:50:53.200 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8912a20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:53.200 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8912a20640 1 -- 172.21.15.155:0/2885178224 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f890c14bfd0 con 0x7f890c0b4df0 2024-08-09T14:50:53.200 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f890a7fc640 4 set_mon_vals no callback set 2024-08-09T14:50:53.200 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8912a20640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:53.200 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8912a20640 10 monclient: _renew_subs 2024-08-09T14:50:53.200 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8912a20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:53.200 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8912a20640 1 -- 172.21.15.155:0/2885178224 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f890c14a400 con 0x7f890c0b4df0 2024-08-09T14:50:53.201 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8912a20640 1 librados: init done 2024-08-09T14:50:53.201 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8ba77fe640 1 --2- 172.21.15.155:0/628073468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ba80b2d80 0x7f8ba8147b70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c00b450 tx=0x7f8b9c00b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:53.201 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8ba4ff9640 10 client.?.objecter ms_handle_connect 0x7f8ba80b2d80 2024-08-09T14:50:53.201 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.193+0000 7f8ba4ff9640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:53.201 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.194+0000 7f8912a20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.194+0000 7f8912a20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.194+0000 7f8912a20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:53.202 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.194+0000 7f8912a20640 1 -- 172.21.15.155:0/2885178224 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f890c0aa9d0 con 0x7f890c0b4df0 2024-08-09T14:50:53.203 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fa27fc640 1 -- 172.21.15.155:0/1916473263 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa4039050 con 0x7f3fac0b3840 2024-08-09T14:50:53.203 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fa27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:53.203 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fa27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:53.203 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fa27fc640 10 monclient: dump: 2024-08-09T14:50:53.203 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:53.204 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.204 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.204 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.204 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:53.204 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.204 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:53.204 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:53.204 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:53.204 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.204 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fa27fc640 10 monclient: _finish_auth 0 2024-08-09T14:50:53.204 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fa27fc640 10 monclient: _check_auth_tickets 2024-08-09T14:50:53.204 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fa27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:53.204 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fa27fc640 1 -- 172.21.15.155:0/1916473263 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3fa4045070 con 0x7f3fac0b3840 2024-08-09T14:50:53.204 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fa27fc640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:53.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fa27fc640 1 -- 172.21.15.155:0/1916473263 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3fa403e870 con 0x7f3fac0b3840 2024-08-09T14:50:53.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fa27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:53.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fa27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:53.205 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fa27fc640 10 monclient: dump: 2024-08-09T14:50:53.205 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:53.205 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.205 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.205 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.205 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:53.205 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.205 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:53.205 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:53.205 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:53.206 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fb3f67640 5 monclient: authenticate success, global_id 4328 2024-08-09T14:50:53.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fb3f67640 10 monclient: _renew_subs 2024-08-09T14:50:53.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fa3fff640 4 set_mon_vals no callback set 2024-08-09T14:50:53.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fb3f67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:53.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.201+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1916473263 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fac1d5460 con 0x7f3fac0b3840 2024-08-09T14:50:53.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.202+0000 7f3fb3f67640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:53.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.202+0000 7f3fb3f67640 10 monclient: _renew_subs 2024-08-09T14:50:53.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.202+0000 7f3fb3f67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:53.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.202+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1916473263 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fac1d5ec0 con 0x7f3fac0b3840 2024-08-09T14:50:53.206 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.202+0000 7f3fb3f67640 1 librados: init done 2024-08-09T14:50:53.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.202+0000 7f8fd37fe640 1 --2- 172.21.15.155:0/1689502221 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fd40b2d80 0x7f8fd40aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:53.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.202+0000 7f8fd37fe640 10 monclient(hunting): get_auth_request con 0x7f8fd40b2d80 auth_method 0 2024-08-09T14:50:53.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.202+0000 7f8fd37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:53.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.202+0000 7f8fd37fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:53.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.202+0000 7f8fd37fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:53.207 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.203+0000 7f8fd37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:53.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.203+0000 7f8fd37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:53.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.203+0000 7f8fd37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:53.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.203+0000 7f3fb3f67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.203+0000 7f3fb3f67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.203+0000 7f3fb3f67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:53.208 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.203+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1916473263 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fac0b3cd0 con 0x7f3fac0b3840 2024-08-09T14:50:53.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.205+0000 7f8fd37fe640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2024-08-09T14:50:53.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.205+0000 7f8fd37fe640 1 -- 172.21.15.155:0/1689502221 >> v1:172.21.15.194:6789/0 conn(0x7f8fd40b4df0 legacy=0x7f8fd413fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.206+0000 7f8fd37fe640 1 -- 172.21.15.155:0/1689502221 >> v1:172.21.15.155:6789/0 conn(0x7f8fd406d8d0 legacy=0x7f8fd4142c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:53.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.206+0000 7f8fd37fe640 10 monclient: _finish_hunting 0 2024-08-09T14:50:53.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.206+0000 7f8fd37fe640 1 monclient: found mon.noname-b 2024-08-09T14:50:53.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.206+0000 7f8fd37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:53.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.206+0000 7f8fd37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:53.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.206+0000 7f8fd37fe640 1 -- 172.21.15.155:0/1689502221 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd4150510 con 0x7f8fd40b2d80 2024-08-09T14:50:53.209 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.208+0000 7f8fd37fe640 1 --2- 172.21.15.155:0/1689502221 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fd40b2d80 0x7f8fd40aa9d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80097d0 tx=0x7f8fc802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8259a7198e6eab1f server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:53.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.234+0000 7f0fef7fe640 1 -- 172.21.15.155:0/3040544002 <== mon.0 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f0ffc05a1a0 con 0x7f10080b22e0 2024-08-09T14:50:53.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.234+0000 7f0fef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.237 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.234+0000 7f0fef7fe640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:53.353 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.351+0000 7f84d6bee640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-09T14:50:53.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.351+0000 7f84d6bee640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-09T14:50:53.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.351+0000 7f84d6bee640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:53.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.351+0000 7f84d6bee640 1 -- 172.21.15.155:0/1379318804 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f84d00b6950 con 0x7f84d00b2d80 2024-08-09T14:50:53.445 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.444+0000 7f8908ff9640 1 -- 172.21.15.155:0/2885178224 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8904047020 con 0x7f890c0b4df0 2024-08-09T14:50:53.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.444+0000 7f8908ff9640 1 --2- 172.21.15.155:0/2885178224 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f88f806d810 0x7f88f806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.445+0000 7f890bfff640 1 --2- 172.21.15.155:0/2885178224 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f88f806d810 0x7f88f806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:53.446 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.445+0000 7f890bfff640 10 monclient: get_auth_request con 0x7f88f806d810 auth_method 0 2024-08-09T14:50:53.447 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.445+0000 7f8908ff9640 10 client.4360.objecter ms_handle_connect 0x7f88f806d810 2024-08-09T14:50:53.447 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.445+0000 7f890bfff640 1 --2- 172.21.15.155:0/2885178224 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f88f806d810 0x7f88f806fcd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f88fc000f60 tx=0x7f88fc007a80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:53.448 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.447+0000 7f8908ff9640 1 -- 172.21.15.155:0/2885178224 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f890402cc00 con 0x7f890c0b4df0 2024-08-09T14:50:53.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.447+0000 7f8908ff9640 10 client.4360.objecter ms_dispatch 0x7f890c000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:53.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.447+0000 7f8908ff9640 3 client.4360.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:53.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.447+0000 7f8908ff9640 3 client.4360.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:53.449 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.447+0000 7f8908ff9640 20 client.4360.objecter dump_active .. 0 homeless 2024-08-09T14:50:53.458 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.457+0000 7f8908ff9640 1 -- 172.21.15.155:0/2885178224 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f890404e030 con 0x7f890c0b4df0 2024-08-09T14:50:53.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.457+0000 7f8908ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.459 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.457+0000 7f8908ff9640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:53.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8ba4ff9640 1 -- 172.21.15.155:0/628073468 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c015020 con 0x7f8ba80b2d80 2024-08-09T14:50:53.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8ba4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:53.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8ba4ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:53.460 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8ba4ff9640 10 monclient: dump: 2024-08-09T14:50:53.460 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:53.460 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.460 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.460 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.460 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:53.460 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.460 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:53.460 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:53.461 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:53.461 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8ba4ff9640 10 monclient: _finish_auth 0 2024-08-09T14:50:53.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8ba4ff9640 10 monclient: _check_auth_tickets 2024-08-09T14:50:53.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8ba4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:53.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8ba4ff9640 1 -- 172.21.15.155:0/628073468 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b9c011890 con 0x7f8ba80b2d80 2024-08-09T14:50:53.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8bae5fa640 5 monclient: authenticate success, global_id 4363 2024-08-09T14:50:53.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8bae5fa640 10 monclient: _renew_subs 2024-08-09T14:50:53.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8bae5fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:53.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8bae5fa640 1 -- 172.21.15.155:0/628073468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba8147300 con 0x7f8ba80b2d80 2024-08-09T14:50:53.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8ba4ff9640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:53.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8ba4ff9640 1 -- 172.21.15.155:0/628073468 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c011c80 con 0x7f8ba80b2d80 2024-08-09T14:50:53.461 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8ba4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:53.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8ba4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:53.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8ba4ff9640 10 monclient: dump: 2024-08-09T14:50:53.462 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:53.462 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.462 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.462 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.462 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:53.462 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.462 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:53.462 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:53.462 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:53.462 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.462 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f8ba67fc640 4 set_mon_vals no callback set 2024-08-09T14:50:53.463 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f819d913640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-09T14:50:53.463 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f819d913640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-09T14:50:53.463 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f819d913640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:53.463 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.458+0000 7f819d913640 1 -- 172.21.15.155:0/2390153102 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8198141ba0 con 0x7f81980b26e0 2024-08-09T14:50:53.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.459+0000 7f8bae5fa640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:53.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.459+0000 7f8bae5fa640 10 monclient: _renew_subs 2024-08-09T14:50:53.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.459+0000 7f8bae5fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:53.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.459+0000 7f8bae5fa640 1 -- 172.21.15.155:0/628073468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba814aba0 con 0x7f8ba80b2d80 2024-08-09T14:50:53.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.459+0000 7f8bae5fa640 1 librados: init done 2024-08-09T14:50:53.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.459+0000 7f8bae5fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.460+0000 7f8bae5fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.460+0000 7f8bae5fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:53.464 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.460+0000 7f8bae5fa640 1 -- 172.21.15.155:0/628073468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba80aa9d0 con 0x7f8ba80b2d80 2024-08-09T14:50:53.475 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fd2ffd640 1 -- 172.21.15.155:0/1689502221 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8035020 con 0x7f8fd40b2d80 2024-08-09T14:50:53.476 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:53.476 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fd2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:53.476 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fd2ffd640 10 monclient: dump: 2024-08-09T14:50:53.476 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:53.476 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.476 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.476 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.476 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:53.476 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.476 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:53.476 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:53.476 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:53.476 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fd2ffd640 10 monclient: _finish_auth 0 2024-08-09T14:50:53.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fd2ffd640 10 monclient: _check_auth_tickets 2024-08-09T14:50:53.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:53.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fd2ffd640 1 -- 172.21.15.155:0/1689502221 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fc8031810 con 0x7f8fd40b2d80 2024-08-09T14:50:53.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fd2ffd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:53.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fd2ffd640 1 -- 172.21.15.155:0/1689502221 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc8031be0 con 0x7f8fd40b2d80 2024-08-09T14:50:53.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fda9d4640 5 monclient: authenticate success, global_id 4331 2024-08-09T14:50:53.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fda9d4640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:53.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fda9d4640 4 set_mon_vals no callback set 2024-08-09T14:50:53.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:53.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fd2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:53.477 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.474+0000 7f8fd2ffd640 10 monclient: dump: 2024-08-09T14:50:53.477 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:53.478 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.478 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.478 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.478 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:53.478 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.478 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:53.478 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:53.478 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:53.478 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.481 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.475+0000 7f8fda9d4640 10 monclient: shutdown 2024-08-09T14:50:53.481 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.475+0000 7f8fda9d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:53.481 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.475+0000 7f8fda9d4640 1 -- 172.21.15.155:0/1689502221 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fd40b2d80 msgr2=0x7f8fd40aa9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.481 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.475+0000 7f8fda9d4640 1 --2- 172.21.15.155:0/1689502221 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fd40b2d80 0x7f8fd40aa9d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8fc80097d0 tx=0x7f8fc802b8b0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.481 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.475+0000 7f8fda9d4640 1 -- 172.21.15.155:0/1689502221 shutdown_connections 2024-08-09T14:50:53.481 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.475+0000 7f8fda9d4640 1 --2- 172.21.15.155:0/1689502221 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fd40b2d80 0x7f8fd40aa9d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.475+0000 7f8fda9d4640 1 -- 172.21.15.155:0/1689502221 >> 172.21.15.155:0/1689502221 conn(0x7f8fd40b8d70 msgr2=0x7f8fd40b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:53.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.475+0000 7f8fda9d4640 1 -- 172.21.15.155:0/1689502221 shutdown_connections 2024-08-09T14:50:53.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.475+0000 7f8fda9d4640 1 -- 172.21.15.155:0/1689502221 wait complete. 2024-08-09T14:50:53.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.476+0000 7f8fda9d4640 10 monclient: build_initial_monmap 2024-08-09T14:50:53.482 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.476+0000 7f8fda9d4640 10 monclient: monmap: 2024-08-09T14:50:53.482 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:53.482 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:53.482 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:53.483 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:53.483 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:53.483 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.483 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:53.483 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:53.483 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:53.483 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.476+0000 7f8fda9d4640 1 librados: starting msgr at 2024-08-09T14:50:53.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.476+0000 7f8fda9d4640 1 librados: starting objecter 2024-08-09T14:50:53.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.477+0000 7f8fda9d4640 1 Processor -- start 2024-08-09T14:50:53.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.477+0000 7f8fda9d4640 1 -- start start 2024-08-09T14:50:53.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.477+0000 7f8fda9d4640 1 librados: setting wanted keys 2024-08-09T14:50:53.483 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.477+0000 7f8fda9d4640 1 librados: calling monclient init 2024-08-09T14:50:53.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.477+0000 7f8fda9d4640 10 monclient: init 2024-08-09T14:50:53.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:53.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 10 monclient: _add_conns ranks=[0,1,2] 2024-08-09T14:50:53.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fd406d8d0 0x7f8fd41450e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 10 monclient(hunting): picked mon.noname-a con 0x7f8fd406d8d0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:53.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fd40b2d80 0x7f8fd41db940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 10 monclient(hunting): picked mon.noname-b con 0x7f8fd40b2d80 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:53.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fd40b4df0 0x7f8fd41dbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 10 monclient(hunting): picked mon.noname-c con 0x7f8fd40b4df0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:53.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:53.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd414e740 con 0x7f8fd406d8d0 2024-08-09T14:50:53.484 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:53.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd41408d0 con 0x7f8fd40b2d80 2024-08-09T14:50:53.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:53.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f8fd414e8c0 con 0x7f8fd40b4df0 2024-08-09T14:50:53.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:53.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fda9d4640 10 monclient(hunting): authenticate will time out at 1260.515408s 2024-08-09T14:50:53.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.478+0000 7f8fd3fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fd406d8d0 0x7f8fd41450e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:53.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.479+0000 7f8fd3fff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fd406d8d0 0x7f8fd41450e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43744/0 (socket says 172.21.15.155:43744) 2024-08-09T14:50:53.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.479+0000 7f8fd3fff640 1 -- 172.21.15.155:0/3203779343 learned_addr learned my addr 172.21.15.155:0/3203779343 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:53.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.479+0000 7f8fd3fff640 10 monclient(hunting): get_auth_request con 0x7f8fd406d8d0 auth_method 0 2024-08-09T14:50:53.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.479+0000 7f8fd3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:53.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.479+0000 7f8fd3fff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:53.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.479+0000 7f8fd3fff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:53.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.479+0000 7f8fd3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:53.485 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.479+0000 7f8fd3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:53.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.479+0000 7f8fd3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:53.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.480+0000 7f8194ff9640 1 -- 172.21.15.155:0/2390153102 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8198141ba0 con 0x7f81980b26e0 2024-08-09T14:50:53.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.480+0000 7f8194ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-09T14:50:53.486 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.480+0000 7f8194ff9640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:53.487 INFO:teuthology.orchestra.run.smithi155.stdout:85899345929 2024-08-09T14:50:53.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 10 librados: watch_flush enter 2024-08-09T14:50:53.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 10 librados: watch_flush exit 2024-08-09T14:50:53.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 20 client.4325.objecter shutdown clearing up homeless session... 2024-08-09T14:50:53.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 10 client.4325.objecter successfully canceled tick 2024-08-09T14:50:53.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 1 -- 172.21.15.155:0/2390153102 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f817806d810 msgr2=0x7f817806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 1 --2- 172.21.15.155:0/2390153102 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f817806d810 0x7f817806fcd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8188009670 tx=0x7f8188006bf0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 10 monclient: shutdown 2024-08-09T14:50:53.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:53.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 1 -- 172.21.15.155:0/2390153102 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81980b26e0 msgr2=0x7f8198144a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 1 --2- 172.21.15.155:0/2390153102 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81980b26e0 0x7f8198144a60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8190009e50 tx=0x7f8190004a40 comp rx=0 tx=0).stop 2024-08-09T14:50:53.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 1 -- 172.21.15.155:0/2390153102 shutdown_connections 2024-08-09T14:50:53.488 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 1 --2- 172.21.15.155:0/2390153102 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f817806d810 0x7f817806fcd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 1 --2- 172.21.15.155:0/2390153102 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f81980b26e0 0x7f8198144a60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 1 --2- 172.21.15.155:0/2390153102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f81980b22e0 0x7f8198144520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 1 --2- 172.21.15.155:0/2390153102 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f81980b0160 0x7f819813ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 1 -- 172.21.15.155:0/2390153102 >> 172.21.15.155:0/2390153102 conn(0x7f819806dbd0 msgr2=0x7f81981433a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:53.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 1 -- 172.21.15.155:0/2390153102 shutdown_connections 2024-08-09T14:50:53.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 1 -- 172.21.15.155:0/2390153102 wait complete. 2024-08-09T14:50:53.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f8fd3fff640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2024-08-09T14:50:53.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f8fd3fff640 1 -- 172.21.15.155:0/3203779343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fd40b4df0 msgr2=0x7f8fd41dbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f8fd3fff640 1 --2- 172.21.15.155:0/3203779343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fd40b4df0 0x7f8fd41dbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.489 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f8fd3fff640 1 -- 172.21.15.155:0/3203779343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fd40b2d80 msgr2=0x7f8fd41db940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.490 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f8fd3fff640 1 --2- 172.21.15.155:0/3203779343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fd40b2d80 0x7f8fd41db940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.490 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f8fd3fff640 10 monclient: _finish_hunting 0 2024-08-09T14:50:53.490 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f8fd3fff640 1 monclient: found mon.noname-a 2024-08-09T14:50:53.490 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f8fd3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:53.490 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f8fd3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:53.490 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f8fd3fff640 1 -- 172.21.15.155:0/3203779343 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd41dc3c0 con 0x7f8fd406d8d0 2024-08-09T14:50:53.490 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.483+0000 7f819d913640 1 librados: shutdown 2024-08-09T14:50:53.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.486+0000 7f8fd3fff640 1 --2- 172.21.15.155:0/3203779343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fd406d8d0 0x7f8fd41450e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400c410 tx=0x7f8fc400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:53.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.486+0000 7f8fd0ff9640 10 client.?.objecter ms_handle_connect 0x7f8fd406d8d0 2024-08-09T14:50:53.491 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.486+0000 7f8fd0ff9640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:53.493 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fd0ff9640 1 -- 172.21.15.155:0/3203779343 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4017070 con 0x7f8fd406d8d0 2024-08-09T14:50:53.494 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fd0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:53.494 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fd0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:53.494 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fd0ff9640 10 monclient: dump: 2024-08-09T14:50:53.494 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:53.494 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.494 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.494 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.494 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:53.495 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.495 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:53.495 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:53.495 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:53.495 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fd0ff9640 10 monclient: _finish_auth 0 2024-08-09T14:50:53.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fd0ff9640 10 monclient: _check_auth_tickets 2024-08-09T14:50:53.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:53.495 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fd0ff9640 1 -- 172.21.15.155:0/3203779343 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fc40035f0 con 0x7f8fd406d8d0 2024-08-09T14:50:53.496 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fd0ff9640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:53.496 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fd0ff9640 1 -- 172.21.15.155:0/3203779343 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4003a20 con 0x7f8fd406d8d0 2024-08-09T14:50:53.496 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fd0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:53.496 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fd0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:53.496 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fd0ff9640 10 monclient: dump: 2024-08-09T14:50:53.496 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:53.496 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:53.497 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.497 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:53.497 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:53.497 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:53.497 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:53.497 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:53.497 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:53.497 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:53.497 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fd27fc640 4 set_mon_vals no callback set 2024-08-09T14:50:53.497 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fda9d4640 5 monclient: authenticate success, global_id 4374 2024-08-09T14:50:53.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fda9d4640 10 monclient: _renew_subs 2024-08-09T14:50:53.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fda9d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:53.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.491+0000 7f8fda9d4640 1 -- 172.21.15.155:0/3203779343 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fd413aa20 con 0x7f8fd406d8d0 2024-08-09T14:50:53.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.492+0000 7f8fda9d4640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:53.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.492+0000 7f8fda9d4640 10 monclient: _renew_subs 2024-08-09T14:50:53.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.492+0000 7f8fda9d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:53.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.492+0000 7f8fda9d4640 1 -- 172.21.15.155:0/3203779343 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fd413d8a0 con 0x7f8fd406d8d0 2024-08-09T14:50:53.498 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.492+0000 7f8fda9d4640 1 librados: init done 2024-08-09T14:50:53.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.493+0000 7f8fda9d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.493+0000 7f8fda9d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.493+0000 7f8fda9d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:53.499 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.493+0000 7f8fda9d4640 1 -- 172.21.15.155:0/3203779343 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fd413a830 con 0x7f8fd406d8d0 2024-08-09T14:50:53.500 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345930 got 85899345929 for osd.2 2024-08-09T14:50:53.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.498+0000 7f3fa27fc640 1 -- 172.21.15.155:0/1916473263 <== mon.2 v2:172.21.15.155:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3fa403ea10 con 0x7f3fac0b3840 2024-08-09T14:50:53.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.499+0000 7f3fa27fc640 1 --2- 172.21.15.155:0/1916473263 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f3f9006d7c0 0x7f3f9006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.499+0000 7f3fb1d03640 1 --2- 172.21.15.155:0/1916473263 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f3f9006d7c0 0x7f3f9006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:53.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.499+0000 7f3fb1d03640 10 monclient: get_auth_request con 0x7f3f9006d7c0 auth_method 0 2024-08-09T14:50:53.501 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.499+0000 7f3fa27fc640 10 client.4328.objecter ms_handle_connect 0x7f3f9006d7c0 2024-08-09T14:50:53.502 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.499+0000 7f3fb1d03640 1 --2- 172.21.15.155:0/1916473263 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f3f9006d7c0 0x7f3f9006fc80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8001770 tx=0x7f3fa8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:53.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.514+0000 7f8fd0ff9640 1 -- 172.21.15.155:0/3203779343 <== mon.0 v2:172.21.15.155:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8fc4003bc0 con 0x7f8fd406d8d0 2024-08-09T14:50:53.515 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.514+0000 7f8fd0ff9640 1 --2- 172.21.15.155:0/3203779343 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f8fc006d7c0 0x7f8fc006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.516 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.515+0000 7f8fd37fe640 1 --2- 172.21.15.155:0/3203779343 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f8fc006d7c0 0x7f8fc006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:53.516 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.515+0000 7f8fd37fe640 10 monclient: get_auth_request con 0x7f8fc006d7c0 auth_method 0 2024-08-09T14:50:53.516 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.515+0000 7f8fd0ff9640 10 client.4374.objecter ms_handle_connect 0x7f8fc006d7c0 2024-08-09T14:50:53.516 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.515+0000 7f8fd37fe640 1 --2- 172.21.15.155:0/3203779343 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f8fc006d7c0 0x7f8fc006fc80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8036000 tx=0x7f8fc8030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:53.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.518+0000 7f8fd0ff9640 1 -- 172.21.15.155:0/3203779343 <== mon.0 v2:172.21.15.155:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f8fc4056a90 con 0x7f8fd406d8d0 2024-08-09T14:50:53.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.518+0000 7f8fd0ff9640 10 client.4374.objecter ms_dispatch 0x7f8fd4000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:53.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.518+0000 7f8fd0ff9640 3 client.4374.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:53.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.518+0000 7f8fd0ff9640 3 client.4374.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:53.520 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.518+0000 7f8fd0ff9640 20 client.4374.objecter dump_active .. 0 homeless 2024-08-09T14:50:53.541 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.540+0000 7f100d08a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-08-09T14:50:53.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.540+0000 7f100d08a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-08-09T14:50:53.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.540+0000 7f100d08a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:53.542 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.540+0000 7f100d08a640 1 -- 172.21.15.155:0/3040544002 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f1008142620 con 0x7f10080b22e0 2024-08-09T14:50:53.575 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.574+0000 7f3fa27fc640 1 -- 172.21.15.155:0/1916473263 <== mon.2 v2:172.21.15.155:3301/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f3fa407aa70 con 0x7f3fac0b3840 2024-08-09T14:50:53.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.574+0000 7f3fa27fc640 10 client.4328.objecter ms_dispatch 0x7f3fac000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:53.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.574+0000 7f3fa27fc640 3 client.4328.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:53.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.574+0000 7f3fa27fc640 3 client.4328.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:53.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.574+0000 7f3fa27fc640 20 client.4328.objecter dump_active .. 0 homeless 2024-08-09T14:50:53.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.574+0000 7f3fa27fc640 1 -- 172.21.15.155:0/1916473263 <== mon.2 v2:172.21.15.155:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f3fa407d0d0 con 0x7f3fac0b3840 2024-08-09T14:50:53.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.574+0000 7f3fa27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.576 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.574+0000 7f3fa27fc640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:53.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.688+0000 7f8912a20640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-09T14:50:53.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.688+0000 7f8912a20640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-09T14:50:53.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.688+0000 7f8912a20640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:53.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.688+0000 7f8912a20640 1 -- 172.21.15.155:0/2885178224 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f890c13e480 con 0x7f890c0b4df0 2024-08-09T14:50:53.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.764+0000 7f3fb3f67640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-09T14:50:53.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.764+0000 7f3fb3f67640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-09T14:50:53.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.764+0000 7f3fb3f67640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.155:3301/0 2024-08-09T14:50:53.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.764+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1916473263 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f3fac1d5630 con 0x7f3fac0b3840 2024-08-09T14:50:53.769 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.768+0000 7f3fa27fc640 1 -- 172.21.15.155:0/1916473263 <== mon.2 v2:172.21.15.155:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3fa407c9d0 con 0x7f3fac0b3840 2024-08-09T14:50:53.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.768+0000 7f3fa27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-08-09T14:50:53.770 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.768+0000 7f3fa27fc640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:53.770 INFO:teuthology.orchestra.run.smithi155.stdout:60129542152 2024-08-09T14:50:53.772 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.770+0000 7f3fb3f67640 10 librados: watch_flush enter 2024-08-09T14:50:53.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.770+0000 7f3fb3f67640 10 librados: watch_flush exit 2024-08-09T14:50:53.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.770+0000 7f3fb3f67640 20 client.4328.objecter shutdown clearing up homeless session... 2024-08-09T14:50:53.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.770+0000 7f3fb3f67640 10 client.4328.objecter successfully canceled tick 2024-08-09T14:50:53.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.770+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1916473263 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f3f9006d7c0 msgr2=0x7f3f9006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.770+0000 7f3fb3f67640 1 --2- 172.21.15.155:0/1916473263 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f3f9006d7c0 0x7f3f9006fc80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3fa8001770 tx=0x7f3fa8009040 comp rx=0 tx=0).stop 2024-08-09T14:50:53.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.770+0000 7f3fb3f67640 10 monclient: shutdown 2024-08-09T14:50:53.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.770+0000 7f3fb3f67640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:53.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.770+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1916473263 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fac0b3840 msgr2=0x7f3fac1d5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.770+0000 7f3fb3f67640 1 --2- 172.21.15.155:0/1916473263 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fac0b3840 0x7f3fac1d5980 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3fac068420 tx=0x7f3fa4004a40 comp rx=0 tx=0).stop 2024-08-09T14:50:53.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.771+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1916473263 shutdown_connections 2024-08-09T14:50:53.773 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.771+0000 7f3fb3f67640 1 --2- 172.21.15.155:0/1916473263 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f3f9006d7c0 0x7f3f9006fc80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.771+0000 7f3fb3f67640 1 --2- 172.21.15.155:0/1916473263 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f3fac0b3840 0x7f3fac1d5980 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.771+0000 7f3fb3f67640 1 --2- 172.21.15.155:0/1916473263 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f3fac0b0540 0x7f3fac0ae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.771+0000 7f3fb3f67640 1 --2- 172.21.15.155:0/1916473263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fac0b0160 0x7f3fac0ae420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.771+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1916473263 >> 172.21.15.155:0/1916473263 conn(0x7f3fac06dbd0 msgr2=0x7f3fac14ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:53.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.771+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1916473263 shutdown_connections 2024-08-09T14:50:53.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.771+0000 7f3fb3f67640 1 -- 172.21.15.155:0/1916473263 wait complete. 2024-08-09T14:50:53.774 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.771+0000 7f3fb3f67640 1 librados: shutdown 2024-08-09T14:50:53.780 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.780+0000 7f84cd7fa640 1 -- 172.21.15.155:0/1379318804 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f84bc07c490 con 0x7f84d00b2d80 2024-08-09T14:50:53.780 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.780+0000 7f84cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-08-09T14:50:53.780 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.780+0000 7f84cd7fa640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:53.780 INFO:teuthology.orchestra.run.smithi155.stdout:64424509447 2024-08-09T14:50:53.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 10 librados: watch_flush enter 2024-08-09T14:50:53.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 10 librados: watch_flush exit 2024-08-09T14:50:53.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 20 client.4348.objecter shutdown clearing up homeless session... 2024-08-09T14:50:53.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 10 client.4348.objecter successfully canceled tick 2024-08-09T14:50:53.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 1 -- 172.21.15.155:0/1379318804 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f84b406d810 msgr2=0x7f84b406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 1 --2- 172.21.15.155:0/1379318804 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f84b406d810 0x7f84b406fcd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f84c4009710 tx=0x7f84c4006c90 comp rx=0 tx=0).stop 2024-08-09T14:50:53.783 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 10 monclient: shutdown 2024-08-09T14:50:53.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:53.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 1 -- 172.21.15.155:0/1379318804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84d00b2d80 msgr2=0x7f84d01cc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 1 --2- 172.21.15.155:0/1379318804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84d00b2d80 0x7f84d01cc1c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f84bc02b950 tx=0x7f84bc0046c0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 1 -- 172.21.15.155:0/1379318804 shutdown_connections 2024-08-09T14:50:53.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 1 --2- 172.21.15.155:0/1379318804 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f84b406d810 0x7f84b406fcd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 1 --2- 172.21.15.155:0/1379318804 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f84d0142c50 0x7f84d01cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 1 --2- 172.21.15.155:0/1379318804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f84d00b2d80 0x7f84d01cc1c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 1 --2- 172.21.15.155:0/1379318804 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f84d006d8d0 0x7f84d013eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 1 -- 172.21.15.155:0/1379318804 >> 172.21.15.155:0/1379318804 conn(0x7f84d00b8d70 msgr2=0x7f84d00b5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:53.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 1 -- 172.21.15.155:0/1379318804 shutdown_connections 2024-08-09T14:50:53.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 1 -- 172.21.15.155:0/1379318804 wait complete. 2024-08-09T14:50:53.784 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.781+0000 7f84d6bee640 1 librados: shutdown 2024-08-09T14:50:53.785 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.4 2024-08-09T14:50:53.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.782+0000 7f8fd0ff9640 1 -- 172.21.15.155:0/3203779343 <== mon.0 v2:172.21.15.155:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f8fc4020020 con 0x7f8fd406d8d0 2024-08-09T14:50:53.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.782+0000 7f8fd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.786 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.782+0000 7f8fd0ff9640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:53.787 DEBUG:teuthology.parallel:result is None 2024-08-09T14:50:53.787 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.785+0000 7f8908ff9640 1 -- 172.21.15.155:0/2885178224 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f890407c680 con 0x7f890c0b4df0 2024-08-09T14:50:53.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.785+0000 7f8908ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-08-09T14:50:53.788 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.785+0000 7f8908ff9640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:53.788 INFO:teuthology.orchestra.run.smithi155.stdout:55834574855 2024-08-09T14:50:53.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.788+0000 7f8912a20640 10 librados: watch_flush enter 2024-08-09T14:50:53.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.788+0000 7f8912a20640 10 librados: watch_flush exit 2024-08-09T14:50:53.792 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.788+0000 7f8912a20640 20 client.4360.objecter shutdown clearing up homeless session... 2024-08-09T14:50:53.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.789+0000 7f8912a20640 10 client.4360.objecter successfully canceled tick 2024-08-09T14:50:53.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.789+0000 7f8912a20640 1 -- 172.21.15.155:0/2885178224 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f88f806d810 msgr2=0x7f88f806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.789+0000 7f8912a20640 1 --2- 172.21.15.155:0/2885178224 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f88f806d810 0x7f88f806fcd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f88fc000f60 tx=0x7f88fc007a80 comp rx=0 tx=0).stop 2024-08-09T14:50:53.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.789+0000 7f8912a20640 10 monclient: shutdown 2024-08-09T14:50:53.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.789+0000 7f8912a20640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:53.793 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.789+0000 7f8912a20640 1 -- 172.21.15.155:0/2885178224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f890c0b4df0 msgr2=0x7f890c1477c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.789+0000 7f8912a20640 1 --2- 172.21.15.155:0/2885178224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f890c0b4df0 0x7f890c1477c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8904009d90 tx=0x7f8904004a40 comp rx=0 tx=0).stop 2024-08-09T14:50:53.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.790+0000 7f8912a20640 1 -- 172.21.15.155:0/2885178224 shutdown_connections 2024-08-09T14:50:53.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.790+0000 7f8912a20640 1 --2- 172.21.15.155:0/2885178224 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f88f806d810 0x7f88f806fcd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.790+0000 7f8912a20640 1 --2- 172.21.15.155:0/2885178224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f890c0b4df0 0x7f890c1477c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.790+0000 7f8912a20640 1 --2- 172.21.15.155:0/2885178224 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f890c0b2d80 0x7f890c147280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.790+0000 7f8912a20640 1 --2- 172.21.15.155:0/2885178224 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f890c06d8d0 0x7f890c149ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.790+0000 7f8912a20640 1 -- 172.21.15.155:0/2885178224 >> 172.21.15.155:0/2885178224 conn(0x7f890c0b8d70 msgr2=0x7f890c0b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:53.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.790+0000 7f8912a20640 1 -- 172.21.15.155:0/2885178224 shutdown_connections 2024-08-09T14:50:53.794 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.790+0000 7f8912a20640 1 -- 172.21.15.155:0/2885178224 wait complete. 2024-08-09T14:50:53.795 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.790+0000 7f8912a20640 1 librados: shutdown 2024-08-09T14:50:53.801 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.1 2024-08-09T14:50:53.802 DEBUG:teuthology.parallel:result is None 2024-08-09T14:50:53.802 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.801+0000 7f8ba4ff9640 1 -- 172.21.15.155:0/628073468 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f8b9c018020 con 0x7f8ba80b2d80 2024-08-09T14:50:53.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.802+0000 7f8ba4ff9640 1 --2- 172.21.15.155:0/628073468 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f8b9406d810 0x7f8b9406fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:53.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.802+0000 7f8ba7fff640 1 --2- 172.21.15.155:0/628073468 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f8b9406d810 0x7f8b9406fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:53.803 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.802+0000 7f8ba7fff640 10 monclient: get_auth_request con 0x7f8b9406d810 auth_method 0 2024-08-09T14:50:53.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.802+0000 7f8ba4ff9640 10 client.4363.objecter ms_handle_connect 0x7f8b9406d810 2024-08-09T14:50:53.804 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.802+0000 7f8ba7fff640 1 --2- 172.21.15.155:0/628073468 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f8b9406d810 0x7f8b9406fcd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b98036000 tx=0x7f8b98030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:53.805 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2024-08-09T14:50:53.805 DEBUG:teuthology.parallel:result is None 2024-08-09T14:50:53.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.808+0000 7f8ba4ff9640 1 -- 172.21.15.155:0/628073468 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 5266+0+0 (secure 0 0 0) 0x7f8b9c057ab0 con 0x7f8ba80b2d80 2024-08-09T14:50:53.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.808+0000 7f8ba4ff9640 10 client.4363.objecter ms_dispatch 0x7f8ba8000b90 osd_map(27..27 src has 1..27) 2024-08-09T14:50:53.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.808+0000 7f8ba4ff9640 3 client.4363.objecter handle_osd_map got epochs [27,27] > 0 2024-08-09T14:50:53.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.808+0000 7f8ba4ff9640 3 client.4363.objecter handle_osd_map decoding full epoch 27 2024-08-09T14:50:53.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.808+0000 7f8ba4ff9640 20 client.4363.objecter dump_active .. 0 homeless 2024-08-09T14:50:53.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.812+0000 7f0fef7fe640 1 -- 172.21.15.155:0/3040544002 <== mon.0 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0ffc05a1a0 con 0x7f10080b22e0 2024-08-09T14:50:53.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.812+0000 7f0fef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-08-09T14:50:53.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.812+0000 7f0fef7fe640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:53.814 INFO:teuthology.orchestra.run.smithi155.stdout:81604378635 2024-08-09T14:50:53.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 10 librados: watch_flush enter 2024-08-09T14:50:53.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 10 librados: watch_flush exit 2024-08-09T14:50:53.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 20 client.4368.objecter shutdown clearing up homeless session... 2024-08-09T14:50:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 10 client.4368.objecter successfully canceled tick 2024-08-09T14:50:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 1 -- 172.21.15.155:0/3040544002 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f0ff406d7c0 msgr2=0x7f0ff406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 1 --2- 172.21.15.155:0/3040544002 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f0ff406d7c0 0x7f0ff406fc80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8035000 tx=0x7f0ff8008030 comp rx=0 tx=0).stop 2024-08-09T14:50:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 10 monclient: shutdown 2024-08-09T14:50:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 1 -- 172.21.15.155:0/3040544002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10080b22e0 msgr2=0x7f1008141e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 1 --2- 172.21.15.155:0/3040544002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10080b22e0 0x7f1008141e10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0ffc00c430 tx=0x7f0ffc00c8f0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 1 -- 172.21.15.155:0/3040544002 shutdown_connections 2024-08-09T14:50:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 1 --2- 172.21.15.155:0/3040544002 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f0ff406d7c0 0x7f0ff406fc80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 1 --2- 172.21.15.155:0/3040544002 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10080b26e0 0x7f1008134710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 1 --2- 172.21.15.155:0/3040544002 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f10080b22e0 0x7f1008141e10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 1 --2- 172.21.15.155:0/3040544002 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f10080b0160 0x7f1008142950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:53.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 1 -- 172.21.15.155:0/3040544002 >> 172.21.15.155:0/3040544002 conn(0x7f100806dbd0 msgr2=0x7f100813c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:53.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 1 -- 172.21.15.155:0/3040544002 shutdown_connections 2024-08-09T14:50:53.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.814+0000 7f100d08a640 1 -- 172.21.15.155:0/3040544002 wait complete. 2024-08-09T14:50:53.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.815+0000 7f100d08a640 1 librados: shutdown 2024-08-09T14:50:53.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.818+0000 7f8ba4ff9640 1 -- 172.21.15.155:0/628073468 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f8b9c01f020 con 0x7f8ba80b2d80 2024-08-09T14:50:53.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.818+0000 7f8ba4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:53.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:53.818+0000 7f8ba4ff9640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:53.830 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378635 got 81604378635 for osd.7 2024-08-09T14:50:53.830 DEBUG:teuthology.parallel:result is None 2024-08-09T14:50:54.002 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.001+0000 7f8fda9d4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-09T14:50:54.002 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.001+0000 7f8fda9d4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-09T14:50:54.002 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.001+0000 7f8fda9d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.155:3300/0 2024-08-09T14:50:54.002 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.001+0000 7f8fda9d4640 1 -- 172.21.15.155:0/3203779343 --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f8fd40aa9d0 con 0x7f8fd406d8d0 2024-08-09T14:50:54.008 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.007+0000 7f8fd0ff9640 1 -- 172.21.15.155:0/3203779343 <== mon.0 v2:172.21.15.155:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8fc40590f0 con 0x7f8fd406d8d0 2024-08-09T14:50:54.008 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.007+0000 7f8fd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-08-09T14:50:54.008 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.007+0000 7f8fd0ff9640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:54.008 INFO:teuthology.orchestra.run.smithi155.stdout:60129542150 2024-08-09T14:50:54.011 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 10 librados: watch_flush enter 2024-08-09T14:50:54.011 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 10 librados: watch_flush exit 2024-08-09T14:50:54.011 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 20 client.4374.objecter shutdown clearing up homeless session... 2024-08-09T14:50:54.011 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 10 client.4374.objecter successfully canceled tick 2024-08-09T14:50:54.011 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 1 -- 172.21.15.155:0/3203779343 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f8fc006d7c0 msgr2=0x7f8fc006fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 1 --2- 172.21.15.155:0/3203779343 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f8fc006d7c0 0x7f8fc006fc80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8fc8036000 tx=0x7f8fc8030000 comp rx=0 tx=0).stop 2024-08-09T14:50:54.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 10 monclient: shutdown 2024-08-09T14:50:54.012 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:54.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 1 -- 172.21.15.155:0/3203779343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fd406d8d0 msgr2=0x7f8fd41450e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 1 --2- 172.21.15.155:0/3203779343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fd406d8d0 0x7f8fd41450e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8fc400c410 tx=0x7f8fc400c8d0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 1 -- 172.21.15.155:0/3203779343 shutdown_connections 2024-08-09T14:50:54.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 1 --2- 172.21.15.155:0/3203779343 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f8fc006d7c0 0x7f8fc006fc80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.013 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 1 --2- 172.21.15.155:0/3203779343 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8fd40b4df0 0x7f8fd41dbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 1 --2- 172.21.15.155:0/3203779343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fd40b2d80 0x7f8fd41db940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 1 --2- 172.21.15.155:0/3203779343 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8fd406d8d0 0x7f8fd41450e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.009+0000 7f8fda9d4640 1 -- 172.21.15.155:0/3203779343 >> 172.21.15.155:0/3203779343 conn(0x7f8fd40b8d70 msgr2=0x7f8fd40b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:54.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.010+0000 7f8fda9d4640 1 -- 172.21.15.155:0/3203779343 shutdown_connections 2024-08-09T14:50:54.014 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.010+0000 7f8fda9d4640 1 -- 172.21.15.155:0/3203779343 wait complete. 2024-08-09T14:50:54.015 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.010+0000 7f8fda9d4640 1 librados: shutdown 2024-08-09T14:50:54.018 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.018+0000 7f8bae5fa640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-09T14:50:54.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.018+0000 7f8bae5fa640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-09T14:50:54.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.018+0000 7f8bae5fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:54.019 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.018+0000 7f8bae5fa640 1 -- 172.21.15.155:0/628073468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f8ba813de20 con 0x7f8ba80b2d80 2024-08-09T14:50:54.022 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.5 2024-08-09T14:50:54.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.021+0000 7f8ba4ff9640 1 -- 172.21.15.155:0/628073468 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8b9c00d040 con 0x7f8ba80b2d80 2024-08-09T14:50:54.022 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.021+0000 7f8ba4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-08-09T14:50:54.023 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.021+0000 7f8ba4ff9640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:54.023 INFO:teuthology.orchestra.run.smithi155.stdout:90194313227 2024-08-09T14:50:54.023 DEBUG:teuthology.parallel:result is None 2024-08-09T14:50:54.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 10 librados: watch_flush enter 2024-08-09T14:50:54.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 10 librados: watch_flush exit 2024-08-09T14:50:54.025 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 20 client.4363.objecter shutdown clearing up homeless session... 2024-08-09T14:50:54.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 10 client.4363.objecter successfully canceled tick 2024-08-09T14:50:54.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 1 -- 172.21.15.155:0/628073468 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f8b9406d810 msgr2=0x7f8b9406fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 1 --2- 172.21.15.155:0/628073468 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f8b9406d810 0x7f8b9406fcd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b98036000 tx=0x7f8b98030000 comp rx=0 tx=0).stop 2024-08-09T14:50:54.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 10 monclient: shutdown 2024-08-09T14:50:54.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:54.026 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 1 -- 172.21.15.155:0/628073468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ba80b2d80 msgr2=0x7f8ba8147b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 1 --2- 172.21.15.155:0/628073468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ba80b2d80 0x7f8ba8147b70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c00b450 tx=0x7f8b9c00b910 comp rx=0 tx=0).stop 2024-08-09T14:50:54.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 1 -- 172.21.15.155:0/628073468 shutdown_connections 2024-08-09T14:50:54.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 1 --2- 172.21.15.155:0/628073468 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f8b9406d810 0x7f8b9406fcd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 1 --2- 172.21.15.155:0/628073468 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f8ba8142c50 0x7f8ba81480b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 1 --2- 172.21.15.155:0/628073468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ba80b2d80 0x7f8ba8147b70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.027 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 1 --2- 172.21.15.155:0/628073468 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f8ba806d8d0 0x7f8ba814a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 1 -- 172.21.15.155:0/628073468 >> 172.21.15.155:0/628073468 conn(0x7f8ba80b8d70 msgr2=0x7f8ba80b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:54.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 1 -- 172.21.15.155:0/628073468 shutdown_connections 2024-08-09T14:50:54.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 1 -- 172.21.15.155:0/628073468 wait complete. 2024-08-09T14:50:54.028 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.023+0000 7f8bae5fa640 1 librados: shutdown 2024-08-09T14:50:54.035 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313227 got 90194313227 for osd.0 2024-08-09T14:50:54.037 DEBUG:teuthology.parallel:result is None 2024-08-09T14:50:54.198 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-08-09T14:50:54.298 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.297+0000 7feaef47a640 10 monclient: get_monmap_and_config 2024-08-09T14:50:54.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.297+0000 7feaef47a640 10 monclient: build_initial_monmap 2024-08-09T14:50:54.299 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.297+0000 7feaef47a640 10 monclient: monmap: 2024-08-09T14:50:54.299 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:54.299 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:54.300 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:54.297812+0000 2024-08-09T14:50:54.300 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:54.297812+0000 2024-08-09T14:50:54.300 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:54.300 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:54.300 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:54.300 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:54.301 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:54.301 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:54.301 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:54.301 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:54.301 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.298+0000 7feaef47a640 1 Processor -- start 2024-08-09T14:50:54.301 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.298+0000 7feaef47a640 1 -- start start 2024-08-09T14:50:54.302 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.298+0000 7feaef47a640 10 monclient: init 2024-08-09T14:50:54.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.299+0000 7feaef47a640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:54.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.299+0000 7feaef47a640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-08-09T14:50:54.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.299+0000 7feaef47a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feae806d990 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:54.306 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.299+0000 7feaef47a640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7feae806cd80 0x7feae81432d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:54.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.299+0000 7feaef47a640 10 monclient(hunting): picked mon.noname-c con 0x7feae806cd80 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:54.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.299+0000 7feaef47a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feae806dd70 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:54.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.299+0000 7feaef47a640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feae8143810 con 0x7feae806dd70 2024-08-09T14:50:54.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.299+0000 7feaef47a640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feae806e2b0 con 0x7feae806d990 2024-08-09T14:50:54.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.299+0000 7feaef47a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:54.307 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.299+0000 7feaef47a640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7feae80afd10 con 0x7feae806cd80 2024-08-09T14:50:54.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.299+0000 7feaef47a640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:54.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.299+0000 7feaef47a640 10 monclient(hunting): authenticate will time out at 1261.335299s 2024-08-09T14:50:54.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.299+0000 7feaeda17640 1 --1- >> v1:172.21.15.155:6789/0 conn(0x7feae806dd70 0x7feae80af600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.155:6789/0 says I am v1:172.21.15.155:34588/0 (socket says 172.21.15.155:34588) 2024-08-09T14:50:54.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.299+0000 7feaeda17640 1 -- 172.21.15.155:0/3168845479 learned_addr learned my addr 172.21.15.155:0/3168845479 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:54.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.300+0000 7feaeca15640 1 --2- 172.21.15.155:0/3168845479 >> v2:172.21.15.194:3300/0 conn(0x7feae806cd80 0x7feae81432d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:54.308 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.300+0000 7feaeca15640 10 monclient(hunting): get_auth_request con 0x7feae806cd80 auth_method 0 2024-08-09T14:50:54.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.300+0000 7feaeca15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:54.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.300+0000 7feaeca15640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:54.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.300+0000 7feaeca15640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:54.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.301+0000 7feaeca15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:54.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.301+0000 7feaeca15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:54.309 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.301+0000 7feaeca15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:54.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.302+0000 7feadffff640 1 -- 172.21.15.155:0/3168845479 <== mon.0 v1:172.21.15.155:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2013210199 0 0) 0x7feae8143810 con 0x7feae806dd70 2024-08-09T14:50:54.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.302+0000 7feadffff640 10 monclient(hunting): _negotiate 2024-08-09T14:50:54.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.302+0000 7feadffff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:54.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.302+0000 7feadffff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:54.310 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.302+0000 7feadffff640 10 monclient(hunting): my global_id is 4377 2024-08-09T14:50:54.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.302+0000 7feadffff640 1 -- 172.21.15.155:0/3168845479 --> v1:172.21.15.155:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fead4003520 con 0x7feae806dd70 2024-08-09T14:50:54.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.303+0000 7feaeca15640 10 monclient(hunting): handle_auth_done global_id 4366 payload 274 2024-08-09T14:50:54.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.303+0000 7feaeca15640 1 -- 172.21.15.155:0/3168845479 >> v1:172.21.15.194:6789/0 conn(0x7feae806d990 legacy=0x7feae8142bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.303+0000 7feaeca15640 1 -- 172.21.15.155:0/3168845479 >> v1:172.21.15.155:6789/0 conn(0x7feae806dd70 legacy=0x7feae80af600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.303+0000 7feaeca15640 10 monclient: _finish_hunting 0 2024-08-09T14:50:54.311 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.303+0000 7feaeca15640 1 monclient: found mon.noname-c 2024-08-09T14:50:54.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.303+0000 7feaeca15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:54.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.303+0000 7feaeca15640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.194:3300/0 2024-08-09T14:50:54.312 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.303+0000 7feaeca15640 1 -- 172.21.15.155:0/3168845479 --> v2:172.21.15.194:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae80afe90 con 0x7feae806cd80 2024-08-09T14:50:54.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.308+0000 7feaeca15640 1 --2- 172.21.15.155:0/3168845479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae806cd80 0x7feae81432d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fead80097a0 tx=0x7fead802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d10a783b48909a9d server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:54.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.311+0000 7feadffff640 1 -- 172.21.15.155:0/3168845479 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead8035020 con 0x7feae806cd80 2024-08-09T14:50:54.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.311+0000 7feadffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:54.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feadffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:50:54.320 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feadffff640 10 monclient: dump: 2024-08-09T14:50:54.320 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:54.321 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:54.321 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.321 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.321 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:54.321 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:54.322 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:54.322 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:54.322 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:54.322 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:54.322 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feadffff640 10 monclient: _finish_auth 0 2024-08-09T14:50:54.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feadffff640 10 monclient: _check_auth_tickets 2024-08-09T14:50:54.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feadffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:54.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feadffff640 1 -- 172.21.15.155:0/3168845479 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fead8031830 con 0x7feae806cd80 2024-08-09T14:50:54.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feadffff640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:54.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feadffff640 1 -- 172.21.15.155:0/3168845479 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fead8031c00 con 0x7feae806cd80 2024-08-09T14:50:54.323 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feaef47a640 5 monclient: authenticate success, global_id 4366 2024-08-09T14:50:54.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feaef47a640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:54.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feaef47a640 4 set_mon_vals no callback set 2024-08-09T14:50:54.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feadffff640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:54.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feadffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:54.324 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feadffff640 10 monclient: dump: 2024-08-09T14:50:54.325 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:54.325 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:54.325 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.325 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.325 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:54.325 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:54.326 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:54.326 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:54.326 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:54.326 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:54.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feaef47a640 10 monclient: shutdown 2024-08-09T14:50:54.326 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feaef47a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:54.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feaef47a640 1 -- 172.21.15.155:0/3168845479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae806cd80 msgr2=0x7feae81432d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feaef47a640 1 --2- 172.21.15.155:0/3168845479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae806cd80 0x7feae81432d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fead80097a0 tx=0x7fead802b8d0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feaef47a640 1 -- 172.21.15.155:0/3168845479 shutdown_connections 2024-08-09T14:50:54.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feaef47a640 1 --2- 172.21.15.155:0/3168845479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae806cd80 0x7feae81432d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feaef47a640 1 -- 172.21.15.155:0/3168845479 >> 172.21.15.155:0/3168845479 conn(0x7feae8068290 msgr2=0x7feae80b6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:54.327 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feaef47a640 1 -- 172.21.15.155:0/3168845479 shutdown_connections 2024-08-09T14:50:54.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.312+0000 7feaef47a640 1 -- 172.21.15.155:0/3168845479 wait complete. 2024-08-09T14:50:54.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.313+0000 7feaef47a640 10 monclient: build_initial_monmap 2024-08-09T14:50:54.328 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.313+0000 7feaef47a640 10 monclient: monmap: 2024-08-09T14:50:54.328 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:54.328 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:54.329 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:54.329 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:54.329 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:54.329 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:54.329 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:54.329 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:54.330 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:54.330 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:54.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.313+0000 7feaef47a640 1 librados: starting msgr at 2024-08-09T14:50:54.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.313+0000 7feaef47a640 1 librados: starting objecter 2024-08-09T14:50:54.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.313+0000 7feaef47a640 1 Processor -- start 2024-08-09T14:50:54.330 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 1 -- start start 2024-08-09T14:50:54.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 1 librados: setting wanted keys 2024-08-09T14:50:54.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 1 librados: calling monclient init 2024-08-09T14:50:54.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 10 monclient: init 2024-08-09T14:50:54.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:54.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 10 monclient: _add_conns ranks=[1,2,0] 2024-08-09T14:50:54.331 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae806cd80 0x7feae8072480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:54.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 10 monclient(hunting): picked mon.noname-b con 0x7feae806cd80 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:54.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feae806d990 0x7feae81441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:54.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 10 monclient(hunting): picked mon.noname-c con 0x7feae806d990 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:54.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feae806dd70 0x7feae81446e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:54.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 10 monclient(hunting): picked mon.noname-a con 0x7feae806dd70 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:54.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:54.332 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7feae814e6a0 con 0x7feae806dd70 2024-08-09T14:50:54.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:54.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7feae80b04d0 con 0x7feae806cd80 2024-08-09T14:50:54.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:54.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7feae814e820 con 0x7feae806d990 2024-08-09T14:50:54.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:54.333 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.314+0000 7feaef47a640 10 monclient(hunting): authenticate will time out at 1261.350618s 2024-08-09T14:50:54.334 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaeda17640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feae806dd70 0x7feae81446e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:54.334 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaeca15640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feae806d990 0x7feae81441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:54.334 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaed216640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae806cd80 0x7feae8072480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:54.334 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaeda17640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feae806dd70 0x7feae81446e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43746/0 (socket says 172.21.15.155:43746) 2024-08-09T14:50:54.334 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaeda17640 1 -- 172.21.15.155:0/2316717253 learned_addr learned my addr 172.21.15.155:0/2316717253 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:54.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaeda17640 10 monclient(hunting): get_auth_request con 0x7feae806dd70 auth_method 0 2024-08-09T14:50:54.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaeda17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:54.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaeda17640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:54.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaeda17640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:54.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaeca15640 10 monclient(hunting): get_auth_request con 0x7feae806d990 auth_method 0 2024-08-09T14:50:54.335 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaeca15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:54.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaeca15640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:54.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaeca15640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:54.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaed216640 10 monclient(hunting): get_auth_request con 0x7feae806cd80 auth_method 0 2024-08-09T14:50:54.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaed216640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:54.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaed216640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:54.336 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.315+0000 7feaed216640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:54.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.316+0000 7feaed216640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:54.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.316+0000 7feaed216640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:54.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.316+0000 7feaed216640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:54.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.316+0000 7feaeda17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:54.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.316+0000 7feaeda17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:54.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.316+0000 7feaeda17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:54.337 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.316+0000 7feaeca15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:54.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.316+0000 7feaeca15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:54.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.316+0000 7feaeca15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:54.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.318+0000 7feaed216640 10 monclient(hunting): handle_auth_done global_id 4372 payload 824 2024-08-09T14:50:54.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.318+0000 7feaed216640 1 -- 172.21.15.155:0/2316717253 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feae806d990 msgr2=0x7feae81441a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.318+0000 7feaed216640 1 --2- 172.21.15.155:0/2316717253 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feae806d990 0x7feae81441a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.318+0000 7feaed216640 1 -- 172.21.15.155:0/2316717253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feae806dd70 msgr2=0x7feae81446e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.318+0000 7feaed216640 1 --2- 172.21.15.155:0/2316717253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feae806dd70 0x7feae81446e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.318+0000 7feaed216640 10 monclient: _finish_hunting 0 2024-08-09T14:50:54.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.318+0000 7feaed216640 1 monclient: found mon.noname-b 2024-08-09T14:50:54.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.318+0000 7feaed216640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:54.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.318+0000 7feaed216640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:54.338 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.318+0000 7feaed216640 1 -- 172.21.15.155:0/2316717253 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feae8148080 con 0x7feae806cd80 2024-08-09T14:50:54.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.321+0000 7feaed216640 1 --2- 172.21.15.155:0/2316717253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae806cd80 0x7feae8072480 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feae400c390 tx=0x7feae400c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:54.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.321+0000 7feaddffb640 10 client.?.objecter ms_handle_connect 0x7feae806cd80 2024-08-09T14:50:54.340 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.321+0000 7feaddffb640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaddffb640 1 -- 172.21.15.155:0/2316717253 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae4017070 con 0x7feae806cd80 2024-08-09T14:50:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaddffb640 10 monclient: dump: 2024-08-09T14:50:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:54.341 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:54.342 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaddffb640 10 monclient: _finish_auth 0 2024-08-09T14:50:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaddffb640 10 monclient: _check_auth_tickets 2024-08-09T14:50:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaddffb640 1 -- 172.21.15.155:0/2316717253 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feae40052d0 con 0x7feae806cd80 2024-08-09T14:50:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaddffb640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaddffb640 1 -- 172.21.15.155:0/2316717253 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feae4003830 con 0x7feae806cd80 2024-08-09T14:50:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaef47a640 5 monclient: authenticate success, global_id 4372 2024-08-09T14:50:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaef47a640 10 monclient: _renew_subs 2024-08-09T14:50:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaef47a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaef47a640 1 -- 172.21.15.155:0/2316717253 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feae8145a20 con 0x7feae806cd80 2024-08-09T14:50:54.342 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:54.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:54.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feaddffb640 10 monclient: dump: 2024-08-09T14:50:54.343 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:54.343 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:54.343 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.343 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.343 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:54.343 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:54.343 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:54.343 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:54.343 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:54.343 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:54.343 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.324+0000 7feadf7fe640 4 set_mon_vals no callback set 2024-08-09T14:50:54.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.325+0000 7feaef47a640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:54.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.325+0000 7feaef47a640 10 monclient: _renew_subs 2024-08-09T14:50:54.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.325+0000 7feaef47a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:54.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.325+0000 7feaef47a640 1 -- 172.21.15.155:0/2316717253 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feae8145590 con 0x7feae806cd80 2024-08-09T14:50:54.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.325+0000 7feaef47a640 1 librados: init done 2024-08-09T14:50:54.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.325+0000 7feaef47a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:54.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.325+0000 7feaef47a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:54.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.325+0000 7feaef47a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:54.344 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.325+0000 7feaef47a640 1 -- 172.21.15.155:0/2316717253 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feae80af600 con 0x7feae806cd80 2024-08-09T14:50:54.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.343+0000 7feaddffb640 1 -- 172.21.15.155:0/2316717253 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7feae401a070 con 0x7feae806cd80 2024-08-09T14:50:54.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.343+0000 7feaddffb640 1 --2- 172.21.15.155:0/2316717253 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7feacc06d810 0x7feacc06fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:54.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.343+0000 7feaeca15640 1 --2- 172.21.15.155:0/2316717253 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7feacc06d810 0x7feacc06fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:54.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.343+0000 7feaeca15640 10 monclient: get_auth_request con 0x7feacc06d810 auth_method 0 2024-08-09T14:50:54.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.344+0000 7feaeca15640 1 --2- 172.21.15.155:0/2316717253 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7feacc06d810 0x7feacc06fcd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fead8036000 tx=0x7fead8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:54.345 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.344+0000 7feaddffb640 10 client.4372.objecter ms_handle_connect 0x7feacc06d810 2024-08-09T14:50:54.501 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-08-09T14:50:54.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.593+0000 7feaddffb640 1 -- 172.21.15.155:0/2316717253 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5238+0+0 (secure 0 0 0) 0x7feae40569f0 con 0x7feae806cd80 2024-08-09T14:50:54.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.593+0000 7feaddffb640 10 client.4372.objecter ms_dispatch 0x7feae8000b90 osd_map(28..28 src has 1..28) 2024-08-09T14:50:54.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.593+0000 7feaddffb640 3 client.4372.objecter handle_osd_map got epochs [28,28] > 0 2024-08-09T14:50:54.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.593+0000 7feaddffb640 3 client.4372.objecter handle_osd_map decoding full epoch 28 2024-08-09T14:50:54.595 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.594+0000 7feaddffb640 20 client.4372.objecter dump_active .. 0 homeless 2024-08-09T14:50:54.602 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.601+0000 7fa963bb1640 10 monclient: get_monmap_and_config 2024-08-09T14:50:54.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.601+0000 7fa963bb1640 10 monclient: build_initial_monmap 2024-08-09T14:50:54.603 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.601+0000 7fa963bb1640 10 monclient: monmap: 2024-08-09T14:50:54.603 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:54.603 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:54.603 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:54.602868+0000 2024-08-09T14:50:54.603 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:54.602868+0000 2024-08-09T14:50:54.603 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:54.603 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:54.603 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:54.603 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:54.604 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:54.604 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:54.604 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:54.604 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:54.606 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.602+0000 7fa963bb1640 1 Processor -- start 2024-08-09T14:50:54.606 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.602+0000 7fa963bb1640 1 -- start start 2024-08-09T14:50:54.606 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.602+0000 7fa963bb1640 10 monclient: init 2024-08-09T14:50:54.606 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.603+0000 7fa963bb1640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:54.606 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.603+0000 7fa963bb1640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-08-09T14:50:54.606 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.603+0000 7fa963bb1640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa95c070af0 0x7fa95c13dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:54.606 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.603+0000 7fa963bb1640 10 monclient(hunting): picked mon.noname-a con 0x7fa95c070af0 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:54.606 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.603+0000 7fa963bb1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa95c073440 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:54.606 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.603+0000 7fa963bb1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa95c144570 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:54.606 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.603+0000 7fa963bb1640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa95c0b0a50 con 0x7fa95c073440 2024-08-09T14:50:54.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.603+0000 7fa963bb1640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa95c073020 con 0x7fa95c144570 2024-08-09T14:50:54.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.603+0000 7fa963bb1640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:54.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.603+0000 7fa963bb1640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7fa95c0b0d10 con 0x7fa95c070af0 2024-08-09T14:50:54.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.603+0000 7fa963bb1640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:54.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.603+0000 7fa963bb1640 10 monclient(hunting): authenticate will time out at 1261.640342s 2024-08-09T14:50:54.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.603+0000 7fa96194d640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa95c070af0 0x7fa95c13dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:54.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.604+0000 7fa96194d640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fa95c070af0 0x7fa95c13dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43760/0 (socket says 172.21.15.155:43760) 2024-08-09T14:50:54.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.604+0000 7fa96194d640 1 -- 172.21.15.155:0/2928154936 learned_addr learned my addr 172.21.15.155:0/2928154936 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:54.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.604+0000 7fa96194d640 10 monclient(hunting): get_auth_request con 0x7fa95c070af0 auth_method 0 2024-08-09T14:50:54.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.604+0000 7fa96194d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:54.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.604+0000 7fa96194d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:54.607 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.604+0000 7fa96194d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:54.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.605+0000 7fa96194d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:54.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.605+0000 7fa96194d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:54.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.605+0000 7fa96194d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:54.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.606+0000 7feaddffb640 1 -- 172.21.15.155:0/2316717253 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7feae4021020 con 0x7feae806cd80 2024-08-09T14:50:54.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.606+0000 7feaddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:54.608 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.606+0000 7feaddffb640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:54.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.607+0000 7fa96194d640 10 monclient(hunting): handle_auth_done global_id 4383 payload 274 2024-08-09T14:50:54.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.607+0000 7fa96194d640 1 -- 172.21.15.155:0/2928154936 >> v1:172.21.15.194:6789/0 conn(0x7fa95c144570 legacy=0x7fa95c14b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:54.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.607+0000 7fa96194d640 1 -- 172.21.15.155:0/2928154936 >> v1:172.21.15.155:6789/0 conn(0x7fa95c073440 legacy=0x7fa95c143e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.607+0000 7fa96194d640 10 monclient: _finish_hunting 0 2024-08-09T14:50:54.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.607+0000 7fa96194d640 1 monclient: found mon.noname-a 2024-08-09T14:50:54.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.607+0000 7fa96194d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:54.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.607+0000 7fa96194d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:54.609 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.607+0000 7fa96194d640 1 -- 172.21.15.155:0/2928154936 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa95c071de0 con 0x7fa95c070af0 2024-08-09T14:50:54.610 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.609+0000 7fa96194d640 1 --2- 172.21.15.155:0/2928154936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa95c070af0 0x7fa95c13dba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa958009e50 tx=0x7fa95802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238aa5b8b8e1f896 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:54.620 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa96094b640 1 -- 172.21.15.155:0/2928154936 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95802ca80 con 0x7fa95c070af0 2024-08-09T14:50:54.620 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa96094b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:54.620 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa96094b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:54.620 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa96094b640 10 monclient: dump: 2024-08-09T14:50:54.620 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:54.620 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:54.621 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.621 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.621 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:54.621 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:54.621 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:54.621 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:54.621 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:54.621 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:54.621 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa96094b640 10 monclient: _finish_auth 0 2024-08-09T14:50:54.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa96094b640 10 monclient: _check_auth_tickets 2024-08-09T14:50:54.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa96094b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:54.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa96094b640 1 -- 172.21.15.155:0/2928154936 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa95802cc20 con 0x7fa95c070af0 2024-08-09T14:50:54.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa963bb1640 5 monclient: authenticate success, global_id 4383 2024-08-09T14:50:54.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa963bb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:54.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa96094b640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:54.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa963bb1640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:54.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa963bb1640 4 set_mon_vals no callback set 2024-08-09T14:50:54.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa96094b640 1 -- 172.21.15.155:0/2928154936 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9580343f0 con 0x7fa95c070af0 2024-08-09T14:50:54.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa96094b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:54.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa96094b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:54.622 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa96094b640 10 monclient: dump: 2024-08-09T14:50:54.623 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:54.623 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:54.623 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.623 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.623 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:54.623 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:54.623 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:54.623 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:54.623 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:54.623 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:54.628 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa963bb1640 10 monclient: shutdown 2024-08-09T14:50:54.628 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa963bb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:54.628 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa963bb1640 1 -- 172.21.15.155:0/2928154936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa95c070af0 msgr2=0x7fa95c13dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.628 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa963bb1640 1 --2- 172.21.15.155:0/2928154936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa95c070af0 0x7fa95c13dba0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa958009e50 tx=0x7fa95802bee0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.628 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa963bb1640 1 -- 172.21.15.155:0/2928154936 shutdown_connections 2024-08-09T14:50:54.628 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa963bb1640 1 --2- 172.21.15.155:0/2928154936 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa95c070af0 0x7fa95c13dba0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.619+0000 7fa963bb1640 1 -- 172.21.15.155:0/2928154936 >> 172.21.15.155:0/2928154936 conn(0x7fa95c068290 msgr2=0x7fa95c0b6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:54.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.620+0000 7fa963bb1640 1 -- 172.21.15.155:0/2928154936 shutdown_connections 2024-08-09T14:50:54.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.620+0000 7fa963bb1640 1 -- 172.21.15.155:0/2928154936 wait complete. 2024-08-09T14:50:54.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.620+0000 7fa963bb1640 10 monclient: build_initial_monmap 2024-08-09T14:50:54.629 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.620+0000 7fa963bb1640 10 monclient: monmap: 2024-08-09T14:50:54.629 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:54.629 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:54.629 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:54.629 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:54.629 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:54.629 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:54.629 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:54.630 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:54.630 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:54.630 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:54.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 1 librados: starting msgr at 2024-08-09T14:50:54.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 1 librados: starting objecter 2024-08-09T14:50:54.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 1 Processor -- start 2024-08-09T14:50:54.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 1 -- start start 2024-08-09T14:50:54.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 1 librados: setting wanted keys 2024-08-09T14:50:54.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 1 librados: calling monclient init 2024-08-09T14:50:54.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 10 monclient: init 2024-08-09T14:50:54.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:54.630 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 10 monclient: _add_conns ranks=[2,0,1] 2024-08-09T14:50:54.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa95c070af0 0x7fa95c0aafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:54.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 10 monclient(hunting): picked mon.noname-c con 0x7fa95c070af0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:54.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa95c073440 0x7fa95c13f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:54.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 10 monclient(hunting): picked mon.noname-a con 0x7fa95c073440 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:54.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa95c144570 0x7fa95c13fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:54.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 10 monclient(hunting): picked mon.noname-b con 0x7fa95c144570 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:54.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:54.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c13e0e0 con 0x7fa95c073440 2024-08-09T14:50:54.631 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:54.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa95c071120 con 0x7fa95c144570 2024-08-09T14:50:54.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:54.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fa95c13e260 con 0x7fa95c070af0 2024-08-09T14:50:54.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:54.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.621+0000 7fa963bb1640 10 monclient(hunting): authenticate will time out at 1261.658977s 2024-08-09T14:50:54.632 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96114c640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa95c073440 0x7fa95c13f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:54.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96214e640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa95c144570 0x7fa95c13fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:54.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96194d640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa95c070af0 0x7fa95c0aafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:54.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96214e640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa95c144570 0x7fa95c13fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:48112/0 (socket says 172.21.15.155:48112) 2024-08-09T14:50:54.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96214e640 1 -- 172.21.15.155:0/1114924261 learned_addr learned my addr 172.21.15.155:0/1114924261 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:54.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96214e640 10 monclient(hunting): get_auth_request con 0x7fa95c144570 auth_method 0 2024-08-09T14:50:54.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96214e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:54.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96214e640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:54.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96214e640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:54.633 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96114c640 10 monclient(hunting): get_auth_request con 0x7fa95c073440 auth_method 0 2024-08-09T14:50:54.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96114c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:54.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96114c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:54.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96114c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:54.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96194d640 10 monclient(hunting): get_auth_request con 0x7fa95c070af0 auth_method 0 2024-08-09T14:50:54.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96194d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:54.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96194d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:54.634 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.622+0000 7fa96194d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:54.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.623+0000 7fa96214e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:54.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.623+0000 7fa96214e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:54.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.623+0000 7fa96214e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:54.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.623+0000 7fa96114c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:54.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.623+0000 7fa96114c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:54.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.623+0000 7fa96114c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:54.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.623+0000 7fa96194d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:54.635 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.623+0000 7fa96194d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:54.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.623+0000 7fa96194d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:54.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.626+0000 7fa96214e640 10 monclient(hunting): handle_auth_done global_id 4375 payload 824 2024-08-09T14:50:54.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.626+0000 7fa96214e640 1 -- 172.21.15.155:0/1114924261 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa95c070af0 msgr2=0x7fa95c0aafe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.626+0000 7fa96214e640 1 --2- 172.21.15.155:0/1114924261 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa95c070af0 0x7fa95c0aafe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.626+0000 7fa96214e640 1 -- 172.21.15.155:0/1114924261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa95c073440 msgr2=0x7fa95c13f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.626+0000 7fa96214e640 1 --2- 172.21.15.155:0/1114924261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa95c073440 0x7fa95c13f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.626+0000 7fa96214e640 10 monclient: _finish_hunting 0 2024-08-09T14:50:54.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.626+0000 7fa96214e640 1 monclient: found mon.noname-b 2024-08-09T14:50:54.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.626+0000 7fa96214e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:54.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.626+0000 7fa96214e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:54.636 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.626+0000 7fa96214e640 1 -- 172.21.15.155:0/1114924261 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa95c0ace10 con 0x7fa95c144570 2024-08-09T14:50:54.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.628+0000 7fa96214e640 1 --2- 172.21.15.155:0/1114924261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa95c144570 0x7fa95c13fe30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa95400b490 tx=0x7fa95400b950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:54.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.628+0000 7fa9527fc640 10 client.?.objecter ms_handle_connect 0x7fa95c144570 2024-08-09T14:50:54.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.628+0000 7fa9527fc640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:54.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa9527fc640 1 -- 172.21.15.155:0/1114924261 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa954015020 con 0x7fa95c144570 2024-08-09T14:50:54.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa9527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:54.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa9527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:54.638 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa9527fc640 10 monclient: dump: 2024-08-09T14:50:54.639 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:54.639 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:54.639 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.639 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.639 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:54.639 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:54.639 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:54.639 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:54.639 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:54.639 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:54.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa9527fc640 10 monclient: _finish_auth 0 2024-08-09T14:50:54.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa9527fc640 10 monclient: _check_auth_tickets 2024-08-09T14:50:54.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa9527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:54.639 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa9527fc640 1 -- 172.21.15.155:0/1114924261 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa954011850 con 0x7fa95c144570 2024-08-09T14:50:54.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa9527fc640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:54.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa9527fc640 1 -- 172.21.15.155:0/1114924261 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa954011c40 con 0x7fa95c144570 2024-08-09T14:50:54.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa9527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:54.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa9527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:54.640 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa9527fc640 10 monclient: dump: 2024-08-09T14:50:54.640 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:54.640 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:54.640 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.640 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:54.640 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:54.640 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:54.640 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:54.640 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa963bb1640 5 monclient: authenticate success, global_id 4375 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa963bb1640 10 monclient: _renew_subs 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa963bb1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa963bb1640 1 -- 172.21.15.155:0/1114924261 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa95c0ad0f0 con 0x7fa95c144570 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa953fff640 4 set_mon_vals no callback set 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa963bb1640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa963bb1640 10 monclient: _renew_subs 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa963bb1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa963bb1640 1 -- 172.21.15.155:0/1114924261 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa95c0ac2a0 con 0x7fa95c144570 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.632+0000 7fa963bb1640 1 librados: init done 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.633+0000 7fa963bb1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.633+0000 7fa963bb1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:54.641 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.633+0000 7fa963bb1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:54.642 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.633+0000 7fa963bb1640 1 -- 172.21.15.155:0/1114924261 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa95c13dba0 con 0x7fa95c144570 2024-08-09T14:50:54.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.651+0000 7fa9527fc640 1 -- 172.21.15.155:0/1114924261 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fa954018020 con 0x7fa95c144570 2024-08-09T14:50:54.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.652+0000 7fa9527fc640 1 --2- 172.21.15.155:0/1114924261 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa94006d810 0x7fa94006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:54.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.652+0000 7fa96194d640 1 --2- 172.21.15.155:0/1114924261 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa94006d810 0x7fa94006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:54.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.652+0000 7fa96194d640 10 monclient: get_auth_request con 0x7fa94006d810 auth_method 0 2024-08-09T14:50:54.653 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.652+0000 7fa9527fc640 10 client.4375.objecter ms_handle_connect 0x7fa94006d810 2024-08-09T14:50:54.654 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.652+0000 7fa96194d640 1 --2- 172.21.15.155:0/1114924261 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa94006d810 0x7fa94006fcd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa958035000 tx=0x7fa958030000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:54.656 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.654+0000 7fa9527fc640 1 -- 172.21.15.155:0/1114924261 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5238+0+0 (secure 0 0 0) 0x7fa954011de0 con 0x7fa95c144570 2024-08-09T14:50:54.656 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.654+0000 7fa9527fc640 10 client.4375.objecter ms_dispatch 0x7fa95c000b90 osd_map(28..28 src has 1..28) 2024-08-09T14:50:54.656 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.654+0000 7fa9527fc640 3 client.4375.objecter handle_osd_map got epochs [28,28] > 0 2024-08-09T14:50:54.656 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.654+0000 7fa9527fc640 3 client.4375.objecter handle_osd_map decoding full epoch 28 2024-08-09T14:50:54.656 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.655+0000 7fa9527fc640 20 client.4375.objecter dump_active .. 0 homeless 2024-08-09T14:50:54.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.788+0000 7feaef47a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-08-09T14:50:54.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.788+0000 7feaef47a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-08-09T14:50:54.789 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.788+0000 7feaef47a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:54.790 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.788+0000 7feaef47a640 1 -- 172.21.15.155:0/2316717253 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7feae8068960 con 0x7feae806cd80 2024-08-09T14:50:54.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.890+0000 7fa9527fc640 1 -- 172.21.15.155:0/1114924261 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7fa95401f020 con 0x7fa95c144570 2024-08-09T14:50:54.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.890+0000 7fa9527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:54.891 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.890+0000 7fa9527fc640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:54.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.894+0000 7feaddffb640 1 -- 172.21.15.155:0/2316717253 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7feae4058950 con 0x7feae806cd80 2024-08-09T14:50:54.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.894+0000 7feaddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-08-09T14:50:54.895 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.894+0000 7feaddffb640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:54.895 INFO:teuthology.orchestra.run.smithi155.stdout:60129542152 2024-08-09T14:50:54.898 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 10 librados: watch_flush enter 2024-08-09T14:50:54.898 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 10 librados: watch_flush exit 2024-08-09T14:50:54.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 20 client.4372.objecter shutdown clearing up homeless session... 2024-08-09T14:50:54.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 10 client.4372.objecter successfully canceled tick 2024-08-09T14:50:54.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 1 -- 172.21.15.155:0/2316717253 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7feacc06d810 msgr2=0x7feacc06fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 1 --2- 172.21.15.155:0/2316717253 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7feacc06d810 0x7feacc06fcd0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fead8036000 tx=0x7fead8008030 comp rx=0 tx=0).stop 2024-08-09T14:50:54.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 10 monclient: shutdown 2024-08-09T14:50:54.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:54.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 1 -- 172.21.15.155:0/2316717253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae806cd80 msgr2=0x7feae8072480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:54.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 1 --2- 172.21.15.155:0/2316717253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae806cd80 0x7feae8072480 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7feae400c390 tx=0x7feae400c850 comp rx=0 tx=0).stop 2024-08-09T14:50:54.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 1 -- 172.21.15.155:0/2316717253 shutdown_connections 2024-08-09T14:50:54.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 1 --2- 172.21.15.155:0/2316717253 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7feacc06d810 0x7feacc06fcd0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 1 --2- 172.21.15.155:0/2316717253 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7feae806dd70 0x7feae81446e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.899 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 1 --2- 172.21.15.155:0/2316717253 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7feae806d990 0x7feae81441a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 1 --2- 172.21.15.155:0/2316717253 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feae806cd80 0x7feae8072480 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:54.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 1 -- 172.21.15.155:0/2316717253 >> 172.21.15.155:0/2316717253 conn(0x7feae8068290 msgr2=0x7feae8140010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:54.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 1 -- 172.21.15.155:0/2316717253 shutdown_connections 2024-08-09T14:50:54.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 1 -- 172.21.15.155:0/2316717253 wait complete. 2024-08-09T14:50:54.900 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:54.896+0000 7feaef47a640 1 librados: shutdown 2024-08-09T14:50:54.910 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.6 2024-08-09T14:50:54.910 DEBUG:teuthology.parallel:result is None 2024-08-09T14:50:55.115 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.114+0000 7fa963bb1640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-09T14:50:55.115 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.114+0000 7fa963bb1640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-09T14:50:55.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.114+0000 7fa963bb1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:55.116 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.114+0000 7fa963bb1640 1 -- 172.21.15.155:0/1114924261 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fa95c068960 con 0x7fa95c144570 2024-08-09T14:50:55.118 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.117+0000 7fa9527fc640 1 -- 172.21.15.155:0/1114924261 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa9540595e0 con 0x7fa95c144570 2024-08-09T14:50:55.119 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.117+0000 7fa9527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-08-09T14:50:55.119 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.117+0000 7fa9527fc640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:55.119 INFO:teuthology.orchestra.run.smithi155.stdout:85899345930 2024-08-09T14:50:55.122 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 10 librados: watch_flush enter 2024-08-09T14:50:55.122 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 10 librados: watch_flush exit 2024-08-09T14:50:55.122 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 20 client.4375.objecter shutdown clearing up homeless session... 2024-08-09T14:50:55.122 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 10 client.4375.objecter successfully canceled tick 2024-08-09T14:50:55.122 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 1 -- 172.21.15.155:0/1114924261 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa94006d810 msgr2=0x7fa94006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:55.122 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 1 --2- 172.21.15.155:0/1114924261 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa94006d810 0x7fa94006fcd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa958035000 tx=0x7fa958030000 comp rx=0 tx=0).stop 2024-08-09T14:50:55.122 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 10 monclient: shutdown 2024-08-09T14:50:55.122 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:55.122 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 1 -- 172.21.15.155:0/1114924261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa95c144570 msgr2=0x7fa95c13fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:55.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 1 --2- 172.21.15.155:0/1114924261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa95c144570 0x7fa95c13fe30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa95400b490 tx=0x7fa95400b950 comp rx=0 tx=0).stop 2024-08-09T14:50:55.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 1 -- 172.21.15.155:0/1114924261 shutdown_connections 2024-08-09T14:50:55.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 1 --2- 172.21.15.155:0/1114924261 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fa94006d810 0x7fa94006fcd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 1 --2- 172.21.15.155:0/1114924261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa95c144570 0x7fa95c13fe30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 1 --2- 172.21.15.155:0/1114924261 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fa95c073440 0x7fa95c13f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 1 --2- 172.21.15.155:0/1114924261 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fa95c070af0 0x7fa95c0aafe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 1 -- 172.21.15.155:0/1114924261 >> 172.21.15.155:0/1114924261 conn(0x7fa95c068290 msgr2=0x7fa95c0b53a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:55.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 1 -- 172.21.15.155:0/1114924261 shutdown_connections 2024-08-09T14:50:55.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 1 -- 172.21.15.155:0/1114924261 wait complete. 2024-08-09T14:50:55.123 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.120+0000 7fa963bb1640 1 librados: shutdown 2024-08-09T14:50:55.133 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345930 got 85899345930 for osd.2 2024-08-09T14:50:55.134 DEBUG:teuthology.parallel:result is None 2024-08-09T14:50:55.134 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-09T14:50:55.134 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-09T14:50:55.235 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.234+0000 7f5334b0e640 10 monclient: get_monmap_and_config 2024-08-09T14:50:55.235 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.234+0000 7f5334b0e640 10 monclient: build_initial_monmap 2024-08-09T14:50:55.236 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.234+0000 7f5334b0e640 10 monclient: monmap: 2024-08-09T14:50:55.236 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:55.236 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:55.236 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:55.235355+0000 2024-08-09T14:50:55.236 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:55.235355+0000 2024-08-09T14:50:55.236 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:55.236 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:55.236 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:55.236 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:55.236 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:55.236 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:55.236 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:55.236 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:55.238 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.234+0000 7f5334b0e640 1 Processor -- start 2024-08-09T14:50:55.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.234+0000 7f5334b0e640 1 -- start start 2024-08-09T14:50:55.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.234+0000 7f5334b0e640 10 monclient: init 2024-08-09T14:50:55.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.235+0000 7f5334b0e640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:55.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.235+0000 7f5334b0e640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-08-09T14:50:55.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.235+0000 7f5334b0e640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f533006d8d0 0x7f53300aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:55.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.235+0000 7f5334b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f533006d8d0 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:55.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.235+0000 7f5334b0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53300b2d80 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:55.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.235+0000 7f5334b0e640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f53300b4df0 0x7f533013fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:55.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.235+0000 7f5334b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f53300b4df0 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:55.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.235+0000 7f5334b0e640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5330143360 con 0x7f53300b2d80 2024-08-09T14:50:55.239 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.235+0000 7f5334b0e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:55.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.235+0000 7f5334b0e640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f533006dcb0 con 0x7f533006d8d0 2024-08-09T14:50:55.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.235+0000 7f5334b0e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:55.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.235+0000 7f5334b0e640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f53300b4680 con 0x7f53300b4df0 2024-08-09T14:50:55.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.235+0000 7f5334b0e640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:55.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.235+0000 7f5334b0e640 10 monclient(hunting): authenticate will time out at 1262.272581s 2024-08-09T14:50:55.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.236+0000 7f532e59c640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f533006d8d0 0x7f53300aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:55.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.236+0000 7f532ed9d640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f53300b4df0 0x7f533013fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:55.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.236+0000 7f532e59c640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f533006d8d0 0x7f53300aa9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:43780/0 (socket says 172.21.15.155:43780) 2024-08-09T14:50:55.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.236+0000 7f532e59c640 1 -- 172.21.15.155:0/1632814811 learned_addr learned my addr 172.21.15.155:0/1632814811 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:55.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.236+0000 7f532e59c640 10 monclient(hunting): get_auth_request con 0x7f533006d8d0 auth_method 0 2024-08-09T14:50:55.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.236+0000 7f532e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:55.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.236+0000 7f532e59c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:55.240 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.236+0000 7f532e59c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:55.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.236+0000 7f532ed9d640 10 monclient(hunting): get_auth_request con 0x7f53300b4df0 auth_method 0 2024-08-09T14:50:55.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.236+0000 7f532ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:55.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.236+0000 7f532ed9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:55.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.236+0000 7f532ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:55.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.237+0000 7f532e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:55.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.237+0000 7f532e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:55.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.237+0000 7f532e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:55.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.237+0000 7f532ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:55.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.237+0000 7f532ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:55.241 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.237+0000 7f532ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:55.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.238+0000 7f532e59c640 10 monclient(hunting): handle_auth_done global_id 4389 payload 274 2024-08-09T14:50:55.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.238+0000 7f532e59c640 1 -- 172.21.15.155:0/1632814811 >> v2:172.21.15.194:3300/0 conn(0x7f53300b4df0 msgr2=0x7f533013fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:55.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.238+0000 7f532e59c640 1 --2- 172.21.15.155:0/1632814811 >> v2:172.21.15.194:3300/0 conn(0x7f53300b4df0 0x7f533013fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.238+0000 7f532e59c640 1 -- 172.21.15.155:0/1632814811 >> v1:172.21.15.194:6789/0 conn(0x7f53300b2d80 legacy=0x7f5330142c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:55.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.238+0000 7f532e59c640 10 monclient: _finish_hunting 0 2024-08-09T14:50:55.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.238+0000 7f532e59c640 1 monclient: found mon.noname-a 2024-08-09T14:50:55.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.238+0000 7f532e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:55.242 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.238+0000 7f532e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:55.243 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.238+0000 7f532e59c640 1 -- 172.21.15.155:0/1632814811 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5330140530 con 0x7f533006d8d0 2024-08-09T14:50:55.243 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.240+0000 7f532e59c640 1 --2- 172.21.15.155:0/1632814811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f533006d8d0 0x7f53300aa9d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5330075f30 tx=0x7f532002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bdfc15b2ea12c33 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:55.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.242+0000 7f532dd9b640 1 -- 172.21.15.155:0/1632814811 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5320034070 con 0x7f533006d8d0 2024-08-09T14:50:55.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.242+0000 7f532dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:55.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.242+0000 7f532dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:55.244 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.242+0000 7f532dd9b640 10 monclient: dump: 2024-08-09T14:50:55.244 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:55.244 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:55.244 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.244 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.244 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:55.244 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:55.244 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:55.244 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:55.245 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:55.245 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:55.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.242+0000 7f532dd9b640 10 monclient: _finish_auth 0 2024-08-09T14:50:55.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.242+0000 7f532dd9b640 10 monclient: _check_auth_tickets 2024-08-09T14:50:55.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.242+0000 7f532dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:55.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.242+0000 7f5334b0e640 5 monclient: authenticate success, global_id 4389 2024-08-09T14:50:55.245 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.242+0000 7f5334b0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-08-09T14:50:55.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.244+0000 7f532dd9b640 1 -- 172.21.15.155:0/1632814811 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5320030860 con 0x7f533006d8d0 2024-08-09T14:50:55.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.244+0000 7f532dd9b640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:55.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.244+0000 7f532dd9b640 1 -- 172.21.15.155:0/1632814811 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5320030c30 con 0x7f533006d8d0 2024-08-09T14:50:55.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.244+0000 7f532dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:55.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.244+0000 7f532dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:55.246 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.244+0000 7f532dd9b640 10 monclient: dump: 2024-08-09T14:50:55.246 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:55.246 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:55.247 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.247 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.247 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:55.247 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:55.247 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:55.247 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:55.247 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:55.247 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:55.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.244+0000 7f5334b0e640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:55.247 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.244+0000 7f5334b0e640 4 set_mon_vals no callback set 2024-08-09T14:50:55.250 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.245+0000 7f5334b0e640 10 monclient: shutdown 2024-08-09T14:50:55.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.245+0000 7f5334b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:55.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.245+0000 7f5334b0e640 1 -- 172.21.15.155:0/1632814811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f533006d8d0 msgr2=0x7f53300aa9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:55.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.245+0000 7f5334b0e640 1 --2- 172.21.15.155:0/1632814811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f533006d8d0 0x7f53300aa9d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5330075f30 tx=0x7f532002b9a0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.245+0000 7f5334b0e640 1 -- 172.21.15.155:0/1632814811 shutdown_connections 2024-08-09T14:50:55.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.245+0000 7f5334b0e640 1 --2- 172.21.15.155:0/1632814811 >> v2:172.21.15.194:3300/0 conn(0x7f53300b4df0 0x7f533013fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.245+0000 7f5334b0e640 1 --2- 172.21.15.155:0/1632814811 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f533006d8d0 0x7f53300aa9d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.245+0000 7f5334b0e640 1 -- 172.21.15.155:0/1632814811 >> 172.21.15.155:0/1632814811 conn(0x7f53300b8d70 msgr2=0x7f53300b9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:55.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.245+0000 7f5334b0e640 1 -- 172.21.15.155:0/1632814811 shutdown_connections 2024-08-09T14:50:55.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.245+0000 7f5334b0e640 1 -- 172.21.15.155:0/1632814811 wait complete. 2024-08-09T14:50:55.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.246+0000 7f5334b0e640 10 monclient: build_initial_monmap 2024-08-09T14:50:55.251 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.246+0000 7f5334b0e640 10 monclient: monmap: 2024-08-09T14:50:55.252 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:55.252 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:55.252 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:55.252 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:55.252 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:55.252 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:55.252 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:55.252 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:55.252 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:55.252 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:55.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.246+0000 7f5334b0e640 1 librados: starting msgr at 2024-08-09T14:50:55.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.246+0000 7f5334b0e640 1 librados: starting objecter 2024-08-09T14:50:55.252 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.246+0000 7f5334b0e640 1 Processor -- start 2024-08-09T14:50:55.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.246+0000 7f5334b0e640 1 -- start start 2024-08-09T14:50:55.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.246+0000 7f5334b0e640 1 librados: setting wanted keys 2024-08-09T14:50:55.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.246+0000 7f5334b0e640 1 librados: calling monclient init 2024-08-09T14:50:55.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.246+0000 7f5334b0e640 10 monclient: init 2024-08-09T14:50:55.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:55.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-08-09T14:50:55.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f533006d8d0 0x7f5330148e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:55.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 10 monclient(hunting): picked mon.noname-c con 0x7f533006d8d0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:55.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53300b2d80 0x7f5330147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:55.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 10 monclient(hunting): picked mon.noname-b con 0x7f53300b2d80 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:55.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53300b4df0 0x7f53301d8650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:55.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 10 monclient(hunting): picked mon.noname-a con 0x7f53300b4df0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:55.253 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:55.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f53301408a0 con 0x7f53300b4df0 2024-08-09T14:50:55.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:55.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f53300b4960 con 0x7f53300b2d80 2024-08-09T14:50:55.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:55.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f533014d6f0 con 0x7f533006d8d0 2024-08-09T14:50:55.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:55.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5334b0e640 10 monclient(hunting): authenticate will time out at 1262.284487s 2024-08-09T14:50:55.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.247+0000 7f5325d9b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53300b2d80 0x7f5330147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:55.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f532ed9d640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53300b4df0 0x7f53301d8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:55.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f532e59c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f533006d8d0 0x7f5330148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:55.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f5325d9b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53300b2d80 0x7f5330147e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:48130/0 (socket says 172.21.15.155:48130) 2024-08-09T14:50:55.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f5325d9b640 1 -- 172.21.15.155:0/2278995623 learned_addr learned my addr 172.21.15.155:0/2278995623 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:55.254 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f53300b2d80 auth_method 0 2024-08-09T14:50:55.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:55.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:55.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f5325d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:55.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f532e59c640 10 monclient(hunting): get_auth_request con 0x7f533006d8d0 auth_method 0 2024-08-09T14:50:55.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f532e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:55.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f532e59c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:55.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f532e59c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:55.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f532ed9d640 10 monclient(hunting): get_auth_request con 0x7f53300b4df0 auth_method 0 2024-08-09T14:50:55.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f532ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:55.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f532ed9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:55.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f532ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:55.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f5325d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:55.255 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f5325d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:55.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f5325d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:55.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f532e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:55.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.248+0000 7f532e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:55.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.249+0000 7f532e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:55.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.249+0000 7f532ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:55.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.249+0000 7f532ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:55.256 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.249+0000 7f532ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:55.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.251+0000 7f5325d9b640 10 monclient(hunting): handle_auth_done global_id 4381 payload 824 2024-08-09T14:50:55.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.251+0000 7f5325d9b640 1 -- 172.21.15.155:0/2278995623 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f533006d8d0 msgr2=0x7f5330148e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:55.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.251+0000 7f5325d9b640 1 --2- 172.21.15.155:0/2278995623 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f533006d8d0 0x7f5330148e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.251+0000 7f5325d9b640 1 -- 172.21.15.155:0/2278995623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53300b4df0 msgr2=0x7f53301d8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:55.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.251+0000 7f5325d9b640 1 --2- 172.21.15.155:0/2278995623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53300b4df0 0x7f53301d8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.251+0000 7f5325d9b640 10 monclient: _finish_hunting 0 2024-08-09T14:50:55.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.251+0000 7f5325d9b640 1 monclient: found mon.noname-b 2024-08-09T14:50:55.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.251+0000 7f5325d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:55.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.251+0000 7f5325d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:55.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.251+0000 7f5325d9b640 1 -- 172.21.15.155:0/2278995623 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5330149fa0 con 0x7f53300b2d80 2024-08-09T14:50:55.257 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.253+0000 7f5325d9b640 1 --2- 172.21.15.155:0/2278995623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53300b2d80 0x7f5330147e90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f531800c400 tx=0x7f531800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:55.258 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.253+0000 7f53277fe640 10 client.?.objecter ms_handle_connect 0x7f53300b2d80 2024-08-09T14:50:55.258 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.253+0000 7f53277fe640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:55.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f53277fe640 1 -- 172.21.15.155:0/2278995623 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53180052d0 con 0x7f53300b2d80 2024-08-09T14:50:55.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f53277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:55.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f53277fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:55.259 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f53277fe640 10 monclient: dump: 2024-08-09T14:50:55.259 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:55.259 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:55.259 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.259 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.259 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:55.259 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:55.259 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:55.259 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:55.260 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:55.260 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:55.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f53277fe640 10 monclient: _finish_auth 0 2024-08-09T14:50:55.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f53277fe640 10 monclient: _check_auth_tickets 2024-08-09T14:50:55.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f53277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:55.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f5334b0e640 5 monclient: authenticate success, global_id 4381 2024-08-09T14:50:55.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f53277fe640 1 -- 172.21.15.155:0/2278995623 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5318005490 con 0x7f53300b2d80 2024-08-09T14:50:55.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f53277fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:55.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f53277fe640 1 -- 172.21.15.155:0/2278995623 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5318007260 con 0x7f53300b2d80 2024-08-09T14:50:55.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f53277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:55.260 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f53277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:55.261 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f53277fe640 10 monclient: dump: 2024-08-09T14:50:55.261 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:55.261 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:55.261 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.261 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.261 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:55.261 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:55.261 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:55.261 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:55.261 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:55.261 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:55.261 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f532d59a640 4 set_mon_vals no callback set 2024-08-09T14:50:55.261 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f5334b0e640 10 monclient: _renew_subs 2024-08-09T14:50:55.261 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f5334b0e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:55.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f5334b0e640 1 -- 172.21.15.155:0/2278995623 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f533013d940 con 0x7f53300b2d80 2024-08-09T14:50:55.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f5334b0e640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:55.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f5334b0e640 10 monclient: _renew_subs 2024-08-09T14:50:55.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f5334b0e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:55.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f5334b0e640 1 -- 172.21.15.155:0/2278995623 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53301472a0 con 0x7f53300b2d80 2024-08-09T14:50:55.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.257+0000 7f5334b0e640 1 librados: init done 2024-08-09T14:50:55.262 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.258+0000 7f5334b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:55.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.258+0000 7f5334b0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:55.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.258+0000 7f5334b0e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:55.263 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.258+0000 7f5334b0e640 1 -- 172.21.15.155:0/2278995623 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53300aa9d0 con 0x7f53300b2d80 2024-08-09T14:50:55.277 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.276+0000 7f53277fe640 1 -- 172.21.15.155:0/2278995623 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5318007400 con 0x7f53300b2d80 2024-08-09T14:50:55.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.276+0000 7f53277fe640 1 --2- 172.21.15.155:0/2278995623 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f531006d810 0x7f531006fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:55.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.276+0000 7f532e59c640 1 --2- 172.21.15.155:0/2278995623 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f531006d810 0x7f531006fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:55.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.276+0000 7f532e59c640 10 monclient: get_auth_request con 0x7f531006d810 auth_method 0 2024-08-09T14:50:55.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.277+0000 7f53277fe640 10 client.4381.objecter ms_handle_connect 0x7f531006d810 2024-08-09T14:50:55.278 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.277+0000 7f532e59c640 1 --2- 172.21.15.155:0/2278995623 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f531006d810 0x7f531006fcd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5320036000 tx=0x7f5320008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:55.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.278+0000 7f53277fe640 1 -- 172.21.15.155:0/2278995623 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5238+0+0 (secure 0 0 0) 0x7f5318056bd0 con 0x7f53300b2d80 2024-08-09T14:50:55.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.278+0000 7f53277fe640 10 client.4381.objecter ms_dispatch 0x7f5330000b90 osd_map(28..28 src has 1..28) 2024-08-09T14:50:55.279 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.278+0000 7f53277fe640 3 client.4381.objecter handle_osd_map got epochs [28,28] > 0 2024-08-09T14:50:55.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.278+0000 7f53277fe640 3 client.4381.objecter handle_osd_map decoding full epoch 28 2024-08-09T14:50:55.280 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.278+0000 7f53277fe640 20 client.4381.objecter dump_active .. 0 homeless 2024-08-09T14:50:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.517+0000 7f53277fe640 1 -- 172.21.15.155:0/2278995623 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f5318058b30 con 0x7f53300b2d80 2024-08-09T14:50:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.517+0000 7f53277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:55.519 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.517+0000 7f53277fe640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:55.690 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.689+0000 7f5334b0e640 1 -- 172.21.15.155:0/2278995623 --> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5330148730 con 0x7f531006d810 2024-08-09T14:50:55.691 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.690+0000 7f53277fe640 1 -- 172.21.15.155:0/2278995623 <== mgr.4103 v2:172.21.15.155:6800/689799774 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35819 (secure 0 0 0) 0x7f5320039450 con 0x7f531006d810 2024-08-09T14:50:55.692 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:50:55.692 INFO:teuthology.orchestra.run.smithi155.stderr:dumped all 2024-08-09T14:50:55.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 10 librados: watch_flush enter 2024-08-09T14:50:55.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 10 librados: watch_flush exit 2024-08-09T14:50:55.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 20 client.4381.objecter shutdown clearing up homeless session... 2024-08-09T14:50:55.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 10 client.4381.objecter successfully canceled tick 2024-08-09T14:50:55.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 1 -- 172.21.15.155:0/2278995623 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f531006d810 msgr2=0x7f531006fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:55.694 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 1 --2- 172.21.15.155:0/2278995623 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f531006d810 0x7f531006fcd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5320036000 tx=0x7f5320008030 comp rx=0 tx=0).stop 2024-08-09T14:50:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 10 monclient: shutdown 2024-08-09T14:50:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 1 -- 172.21.15.155:0/2278995623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53300b2d80 msgr2=0x7f5330147e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 1 --2- 172.21.15.155:0/2278995623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53300b2d80 0x7f5330147e90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f531800c400 tx=0x7f531800c8c0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 1 -- 172.21.15.155:0/2278995623 shutdown_connections 2024-08-09T14:50:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 1 --2- 172.21.15.155:0/2278995623 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f531006d810 0x7f531006fcd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 1 --2- 172.21.15.155:0/2278995623 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f53300b4df0 0x7f53301d8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 1 --2- 172.21.15.155:0/2278995623 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f53300b2d80 0x7f5330147e90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 1 --2- 172.21.15.155:0/2278995623 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f533006d8d0 0x7f5330148e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 1 -- 172.21.15.155:0/2278995623 >> 172.21.15.155:0/2278995623 conn(0x7f53300b8d70 msgr2=0x7f53300b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 1 -- 172.21.15.155:0/2278995623 shutdown_connections 2024-08-09T14:50:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 1 -- 172.21.15.155:0/2278995623 wait complete. 2024-08-09T14:50:55.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.692+0000 7f5334b0e640 1 librados: shutdown 2024-08-09T14:50:55.705 INFO:teuthology.orchestra.run.smithi155.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-08-09T14:50:55.562626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219056,"kb_used_data":2896,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755664,"statfs":{"total":773094113280,"available":772869799936,"internally_reserved":0,"allocated":2965504,"data_stored":1585146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001268"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:47.912680+0000","last_change":"2024-08-09T14:50:47.922102+0000","last_active":"2024-08-09T14:50:47.912680+0000","last_peered":"2024-08-09T14:50:47.912680+0000","last_clean":"2024-08-09T14:50:47.912680+0000","last_became_active":"2024-08-09T14:50:45.816119+0000","last_became_peered":"2024-08-09T14:50:45.816119+0000","last_unstale":"2024-08-09T14:50:47.912680+0000","last_undegraded":"2024-08-09T14:50:47.912680+0000","last_fullsized":"2024-08-09T14:50:47.912680+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-10T19:56:58.452258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.042979287999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:48.067628+0000","last_change":"2024-08-09T14:50:48.104433+0000","last_active":"2024-08-09T14:50:48.067628+0000","last_peered":"2024-08-09T14:50:48.067628+0000","last_clean":"2024-08-09T14:50:48.067628+0000","last_became_active":"2024-08-09T14:50:45.522079+0000","last_became_peered":"2024-08-09T14:50:45.522079+0000","last_unstale":"2024-08-09T14:50:48.067628+0000","last_undegraded":"2024-08-09T14:50:48.067628+0000","last_fullsized":"2024-08-09T14:50:48.067628+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-10T18:50:57.617856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061146998000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:47.921509+0000","last_change":"2024-08-09T14:50:47.925559+0000","last_active":"2024-08-09T14:50:47.921509+0000","last_peered":"2024-08-09T14:50:47.921509+0000","last_clean":"2024-08-09T14:50:47.921509+0000","last_became_active":"2024-08-09T14:50:44.886965+0000","last_became_peered":"2024-08-09T14:50:44.886965+0000","last_unstale":"2024-08-09T14:50:47.921509+0000","last_undegraded":"2024-08-09T14:50:47.921509+0000","last_fullsized":"2024-08-09T14:50:47.921509+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-11T01:19:47.357851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033223413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:48.062813+0000","last_change":"2024-08-09T14:50:48.103202+0000","last_active":"2024-08-09T14:50:48.062813+0000","last_peered":"2024-08-09T14:50:48.062813+0000","last_clean":"2024-08-09T14:50:48.062813+0000","last_became_active":"2024-08-09T14:50:45.500606+0000","last_became_peered":"2024-08-09T14:50:45.500606+0000","last_unstale":"2024-08-09T14:50:48.062813+0000","last_undegraded":"2024-08-09T14:50:48.062813+0000","last_fullsized":"2024-08-09T14:50:48.062813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-11T01:40:33.738679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.072523375000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"27'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:47.905428+0000","last_change":"2024-08-09T14:50:47.905428+0000","last_active":"2024-08-09T14:50:47.905428+0000","last_peered":"2024-08-09T14:50:47.905428+0000","last_clean":"2024-08-09T14:50:47.905428+0000","last_became_active":"2024-08-09T14:50:45.478451+0000","last_became_peered":"2024-08-09T14:50:45.478451+0000","last_unstale":"2024-08-09T14:50:47.905428+0000","last_undegraded":"2024-08-09T14:50:47.905428+0000","last_fullsized":"2024-08-09T14:50:47.905428+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-11T00:22:14.970158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.058190247,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:48.123636+0000","last_change":"2024-08-09T14:50:48.123636+0000","last_active":"2024-08-09T14:50:48.123636+0000","last_peered":"2024-08-09T14:50:48.123636+0000","last_clean":"2024-08-09T14:50:48.123636+0000","last_became_active":"2024-08-09T14:50:44.778689+0000","last_became_peered":"2024-08-09T14:50:44.778689+0000","last_unstale":"2024-08-09T14:50:48.123636+0000","last_undegraded":"2024-08-09T14:50:48.123636+0000","last_fullsized":"2024-08-09T14:50:48.123636+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-10T21:56:49.407967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.093420954,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:47.796120+0000","last_change":"2024-08-09T14:50:47.838899+0000","last_active":"2024-08-09T14:50:47.796120+0000","last_peered":"2024-08-09T14:50:47.796120+0000","last_clean":"2024-08-09T14:50:47.796120+0000","last_became_active":"2024-08-09T14:50:44.798715+0000","last_became_peered":"2024-08-09T14:50:44.798715+0000","last_unstale":"2024-08-09T14:50:47.796120+0000","last_undegraded":"2024-08-09T14:50:47.796120+0000","last_fullsized":"2024-08-09T14:50:47.796120+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-10T18:48:40.847505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.066653635000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"24'1","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:47.888484+0000","last_change":"2024-08-09T14:50:47.901627+0000","last_active":"2024-08-09T14:50:47.888484+0000","last_peered":"2024-08-09T14:50:47.888484+0000","last_clean":"2024-08-09T14:50:47.888484+0000","last_became_active":"2024-08-09T14:50:45.496769+0000","last_became_peered":"2024-08-09T14:50:45.496769+0000","last_unstale":"2024-08-09T14:50:47.888484+0000","last_undegraded":"2024-08-09T14:50:47.888484+0000","last_fullsized":"2024-08-09T14:50:47.888484+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-10T23:34:09.906747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098838583999999993,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"20'76","reported_seq":135,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-09T14:50:54.421600+0000","last_change":"2024-08-09T14:50:32.662579+0000","last_active":"2024-08-09T14:50:54.421600+0000","last_peered":"2024-08-09T14:50:54.421600+0000","last_clean":"2024-08-09T14:50:54.421600+0000","last_became_active":"2024-08-09T14:50:32.629205+0000","last_became_peered":"2024-08-09T14:50:32.629205+0000","last_unstale":"2024-08-09T14:50:54.421600+0000","last_undegraded":"2024-08-09T14:50:54.421600+0000","last_fullsized":"2024-08-09T14:50:54.421600+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:30.526481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:30.526481+0000","last_clean_scrub_stamp":"2024-08-09T14:50:30.526481+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-10T19:45:32.736889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":21,"seq":90194313228,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":102778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":97312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":19,"seq":81604378636,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":97312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":500618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542153,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":97331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":667648,"data_stored":495152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542153,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":97312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":97331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-09T14:50:55.706 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-09T14:50:55.805 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.804+0000 7f5e1c8d6640 10 monclient: get_monmap_and_config 2024-08-09T14:50:55.806 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.804+0000 7f5e1c8d6640 10 monclient: build_initial_monmap 2024-08-09T14:50:55.806 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.804+0000 7f5e1c8d6640 10 monclient: monmap: 2024-08-09T14:50:55.806 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:55.806 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:55.806 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:55.805479+0000 2024-08-09T14:50:55.806 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:55.805479+0000 2024-08-09T14:50:55.806 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:55.806 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:55.806 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:55.806 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:55.806 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:55.807 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:55.807 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:55.807 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:55.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 1 Processor -- start 2024-08-09T14:50:55.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 1 -- start start 2024-08-09T14:50:55.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 10 monclient: init 2024-08-09T14:50:55.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:55.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-08-09T14:50:55.809 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e18067960 0x7f5e180af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:55.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 10 monclient(hunting): picked mon.noname-b con 0x7f5e18067960 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:55.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f5e18143510 0x7f5e18072dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:55.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 10 monclient(hunting): picked mon.noname-c con 0x7f5e18143510 addr v2:172.21.15.194:3300/0 2024-08-09T14:50:55.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5e18073310 0x7f5e18146190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:55.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 10 monclient(hunting): picked mon.noname-a con 0x7f5e18073310 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:55.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:55.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f5e18067d40 con 0x7f5e18073310 2024-08-09T14:50:55.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:55.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f5e180afbe0 con 0x7f5e18143510 2024-08-09T14:50:55.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:55.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f5e18070450 con 0x7f5e18067960 2024-08-09T14:50:55.810 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:55.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.805+0000 7f5e1c8d6640 10 monclient(hunting): authenticate will time out at 1262.842836s 2024-08-09T14:50:55.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e1659c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e18067960 0x7f5e180af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:55.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e16d9d640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f5e18073310 0x7f5e18146190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:55.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e15d9b640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f5e18143510 0x7f5e18072dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:55.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e1659c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e18067960 0x7f5e180af6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:60962/0 (socket says 172.21.15.155:60962) 2024-08-09T14:50:55.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e1659c640 1 -- 172.21.15.155:0/2492062134 learned_addr learned my addr 172.21.15.155:0/2492062134 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:55.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e1659c640 10 monclient(hunting): get_auth_request con 0x7f5e18067960 auth_method 0 2024-08-09T14:50:55.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:55.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e1659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:55.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e1659c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:55.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e15d9b640 10 monclient(hunting): get_auth_request con 0x7f5e18143510 auth_method 0 2024-08-09T14:50:55.811 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e15d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:55.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e15d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:55.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:55.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e16d9d640 10 monclient(hunting): get_auth_request con 0x7f5e18073310 auth_method 0 2024-08-09T14:50:55.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:55.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e16d9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:55.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.806+0000 7f5e16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:55.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.807+0000 7f5e1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:55.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.807+0000 7f5e1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:55.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.807+0000 7f5e1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:55.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.807+0000 7f5e15d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:55.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.807+0000 7f5e15d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:55.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.807+0000 7f5e15d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:55.812 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.807+0000 7f5e16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:55.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.807+0000 7f5e16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:55.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.807+0000 7f5e16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:55.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.808+0000 7f5e1659c640 10 monclient(hunting): handle_auth_done global_id 4343 payload 274 2024-08-09T14:50:55.813 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.808+0000 7f5e1659c640 1 -- 172.21.15.155:0/2492062134 >> v2:172.21.15.194:3300/0 conn(0x7f5e18143510 msgr2=0x7f5e18072dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:55.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.808+0000 7f5e1659c640 1 --2- 172.21.15.155:0/2492062134 >> v2:172.21.15.194:3300/0 conn(0x7f5e18143510 0x7f5e18072dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.808+0000 7f5e1659c640 1 -- 172.21.15.155:0/2492062134 >> v2:172.21.15.155:3300/0 conn(0x7f5e18073310 msgr2=0x7f5e18146190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:50:55.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.808+0000 7f5e1659c640 1 --2- 172.21.15.155:0/2492062134 >> v2:172.21.15.155:3300/0 conn(0x7f5e18073310 0x7f5e18146190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.808+0000 7f5e1659c640 10 monclient: _finish_hunting 0 2024-08-09T14:50:55.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.809+0000 7f5e1659c640 1 monclient: found mon.noname-b 2024-08-09T14:50:55.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.809+0000 7f5e1659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:55.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.809+0000 7f5e1659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.155:3301/0 2024-08-09T14:50:55.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.809+0000 7f5e1659c640 1 -- 172.21.15.155:0/2492062134 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e18140750 con 0x7f5e18067960 2024-08-09T14:50:55.814 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.811+0000 7f5e1659c640 1 --2- 172.21.15.155:0/2492062134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e18067960 0x7f5e180af6a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e180ae990 tx=0x7f5e0802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e120d0b0de518bc server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:55.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.814+0000 7f5e1559a640 1 -- 172.21.15.155:0/2492062134 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e08035020 con 0x7f5e18067960 2024-08-09T14:50:55.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.814+0000 7f5e1559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:55.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.814+0000 7f5e1559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:55.816 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.814+0000 7f5e1559a640 10 monclient: dump: 2024-08-09T14:50:55.816 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:55.816 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:55.816 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.816 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.817 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:55.817 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:55.817 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:55.817 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:55.817 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:55.817 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:55.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.814+0000 7f5e1559a640 10 monclient: _finish_auth 0 2024-08-09T14:50:55.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.814+0000 7f5e1559a640 10 monclient: _check_auth_tickets 2024-08-09T14:50:55.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.814+0000 7f5e1559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:55.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.814+0000 7f5e1559a640 1 -- 172.21.15.155:0/2492062134 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e08031640 con 0x7f5e18067960 2024-08-09T14:50:55.817 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.814+0000 7f5e1559a640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:55.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.814+0000 7f5e1559a640 1 -- 172.21.15.155:0/2492062134 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0802d040 con 0x7f5e18067960 2024-08-09T14:50:55.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.814+0000 7f5e1559a640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:55.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.814+0000 7f5e1559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:50:55.818 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.814+0000 7f5e1559a640 10 monclient: dump: 2024-08-09T14:50:55.818 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:55.818 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:55.818 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.818 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.818 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:55.818 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:55.818 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:55.818 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:55.819 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:55.819 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:55.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.815+0000 7f5e1c8d6640 5 monclient: authenticate success, global_id 4343 2024-08-09T14:50:55.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.815+0000 7f5e1c8d6640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:55.819 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.815+0000 7f5e1c8d6640 4 set_mon_vals no callback set 2024-08-09T14:50:55.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.815+0000 7f5e1c8d6640 10 monclient: shutdown 2024-08-09T14:50:55.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.815+0000 7f5e1c8d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:55.822 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.815+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/2492062134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e18067960 msgr2=0x7f5e180af6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:55.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.815+0000 7f5e1c8d6640 1 --2- 172.21.15.155:0/2492062134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e18067960 0x7f5e180af6a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f5e180ae990 tx=0x7f5e0802b8f0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.815+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/2492062134 shutdown_connections 2024-08-09T14:50:55.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.815+0000 7f5e1c8d6640 1 --2- 172.21.15.155:0/2492062134 >> v2:172.21.15.155:3300/0 conn(0x7f5e18073310 0x7f5e18146190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.815+0000 7f5e1c8d6640 1 --2- 172.21.15.155:0/2492062134 >> v2:172.21.15.194:3300/0 conn(0x7f5e18143510 0x7f5e18072dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.815+0000 7f5e1c8d6640 1 --2- 172.21.15.155:0/2492062134 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e18067960 0x7f5e180af6a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.815+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/2492062134 >> 172.21.15.155:0/2492062134 conn(0x7f5e180b17a0 msgr2=0x7f5e180b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:55.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.815+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/2492062134 shutdown_connections 2024-08-09T14:50:55.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.815+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/2492062134 wait complete. 2024-08-09T14:50:55.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.816+0000 7f5e1c8d6640 10 monclient: build_initial_monmap 2024-08-09T14:50:55.823 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.816+0000 7f5e1c8d6640 10 monclient: monmap: 2024-08-09T14:50:55.824 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:55.824 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:55.824 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:55.824 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:55.824 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:55.824 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:55.824 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:55.824 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:55.824 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:55.824 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:55.824 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 1 librados: starting msgr at 2024-08-09T14:50:55.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 1 librados: starting objecter 2024-08-09T14:50:55.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 1 Processor -- start 2024-08-09T14:50:55.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 1 -- start start 2024-08-09T14:50:55.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 1 librados: setting wanted keys 2024-08-09T14:50:55.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 1 librados: calling monclient init 2024-08-09T14:50:55.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 10 monclient: init 2024-08-09T14:50:55.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:55.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-08-09T14:50:55.825 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e18067960 0x7f5e18144970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:55.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 10 monclient(hunting): picked mon.noname-c con 0x7f5e18067960 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:55.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e18073310 0x7f5e18138e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:55.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 10 monclient(hunting): picked mon.noname-b con 0x7f5e18073310 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:55.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e18143510 0x7f5e18139360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:55.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 10 monclient(hunting): picked mon.noname-a con 0x7f5e18143510 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:55.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:55.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7f5e18142d90 con 0x7f5e18143510 2024-08-09T14:50:55.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:55.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5e18142c10 con 0x7f5e18073310 2024-08-09T14:50:55.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:55.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f5e1806f160 con 0x7f5e18067960 2024-08-09T14:50:55.826 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:55.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.817+0000 7f5e1c8d6640 10 monclient(hunting): authenticate will time out at 1262.854940s 2024-08-09T14:50:55.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.818+0000 7f5e1659c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e18067960 0x7f5e18144970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:55.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.818+0000 7f5e1659c640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e18067960 0x7f5e18144970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:60974/0 (socket says 172.21.15.155:60974) 2024-08-09T14:50:55.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.818+0000 7f5e1659c640 1 -- 172.21.15.155:0/290036674 learned_addr learned my addr 172.21.15.155:0/290036674 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:55.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.818+0000 7f5e1659c640 10 monclient(hunting): get_auth_request con 0x7f5e18067960 auth_method 0 2024-08-09T14:50:55.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.818+0000 7f5e1659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:55.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.818+0000 7f5e1659c640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:55.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.818+0000 7f5e1659c640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:55.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.818+0000 7f5e15d9b640 1 --2- 172.21.15.155:0/290036674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e18073310 0x7f5e18138e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:55.827 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.818+0000 7f5e16d9d640 1 --2- 172.21.15.155:0/290036674 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e18143510 0x7f5e18139360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:55.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e15d9b640 10 monclient(hunting): get_auth_request con 0x7f5e18073310 auth_method 0 2024-08-09T14:50:55.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e15d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:55.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e15d9b640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:55.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:55.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e16d9d640 10 monclient(hunting): get_auth_request con 0x7f5e18143510 auth_method 0 2024-08-09T14:50:55.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e16d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:55.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e16d9d640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:55.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e16d9d640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:55.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e1659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:55.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e1659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:55.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e1659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:55.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e15d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:55.828 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e15d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:55.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e15d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:55.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e16d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:55.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e16d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:55.829 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.819+0000 7f5e16d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:55.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.822+0000 7f5e15d9b640 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2024-08-09T14:50:55.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.822+0000 7f5e15d9b640 1 -- 172.21.15.155:0/290036674 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e18067960 msgr2=0x7f5e18144970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:55.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.822+0000 7f5e15d9b640 1 --2- 172.21.15.155:0/290036674 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e18067960 0x7f5e18144970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.822+0000 7f5e15d9b640 1 -- 172.21.15.155:0/290036674 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e18143510 msgr2=0x7f5e18139360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:55.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.822+0000 7f5e15d9b640 1 --2- 172.21.15.155:0/290036674 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e18143510 0x7f5e18139360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:55.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.822+0000 7f5e15d9b640 10 monclient: _finish_hunting 0 2024-08-09T14:50:55.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.822+0000 7f5e15d9b640 1 monclient: found mon.noname-b 2024-08-09T14:50:55.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.822+0000 7f5e15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:55.831 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.822+0000 7f5e15d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:55.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.822+0000 7f5e15d9b640 1 -- 172.21.15.155:0/290036674 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e181398a0 con 0x7f5e18073310 2024-08-09T14:50:55.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.824+0000 7f5e15d9b640 1 --2- 172.21.15.155:0/290036674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e18073310 0x7f5e18138e20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00e8a0 tx=0x7f5e0c00ed60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:55.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.824+0000 7f5e06ffd640 10 client.?.objecter ms_handle_connect 0x7f5e18073310 2024-08-09T14:50:55.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.824+0000 7f5e06ffd640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:55.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e06ffd640 1 -- 172.21.15.155:0/290036674 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c017070 con 0x7f5e18073310 2024-08-09T14:50:55.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:55.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e06ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:55.832 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e06ffd640 10 monclient: dump: 2024-08-09T14:50:55.832 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:55.832 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:55.832 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.832 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.833 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:55.833 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:55.833 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:55.833 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:55.833 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:55.833 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:55.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e06ffd640 10 monclient: _finish_auth 0 2024-08-09T14:50:55.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e06ffd640 10 monclient: _check_auth_tickets 2024-08-09T14:50:55.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:55.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e06ffd640 1 -- 172.21.15.155:0/290036674 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e0c00c940 con 0x7f5e18073310 2024-08-09T14:50:55.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e1c8d6640 5 monclient: authenticate success, global_id 4387 2024-08-09T14:50:55.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e1c8d6640 10 monclient: _renew_subs 2024-08-09T14:50:55.833 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e1c8d6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:55.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/290036674 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e181467a0 con 0x7f5e18073310 2024-08-09T14:50:55.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e06ffd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:55.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e06ffd640 1 -- 172.21.15.155:0/290036674 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e0c009e60 con 0x7f5e18073310 2024-08-09T14:50:55.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:55.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e06ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:55.834 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e06ffd640 10 monclient: dump: 2024-08-09T14:50:55.834 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:55.834 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:55.834 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.834 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:55.834 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:55.834 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:55.835 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e1c8d6640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e1c8d6640 10 monclient: _renew_subs 2024-08-09T14:50:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e14d99640 4 set_mon_vals no callback set 2024-08-09T14:50:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e1c8d6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/290036674 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e181d5a20 con 0x7f5e18073310 2024-08-09T14:50:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.828+0000 7f5e1c8d6640 1 librados: init done 2024-08-09T14:50:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.829+0000 7f5e1c8d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.829+0000 7f5e1c8d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.829+0000 7f5e1c8d6640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:55.835 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.829+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/290036674 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e18143930 con 0x7f5e18073310 2024-08-09T14:50:55.847 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.846+0000 7f5e06ffd640 1 -- 172.21.15.155:0/290036674 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f5e0c01a070 con 0x7f5e18073310 2024-08-09T14:50:55.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.847+0000 7f5e06ffd640 1 --2- 172.21.15.155:0/290036674 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f5df806d810 0x7f5df806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:55.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.847+0000 7f5e1659c640 1 --2- 172.21.15.155:0/290036674 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f5df806d810 0x7f5df806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:55.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.847+0000 7f5e1659c640 10 monclient: get_auth_request con 0x7f5df806d810 auth_method 0 2024-08-09T14:50:55.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.847+0000 7f5e06ffd640 10 client.4387.objecter ms_handle_connect 0x7f5df806d810 2024-08-09T14:50:55.848 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.847+0000 7f5e1659c640 1 --2- 172.21.15.155:0/290036674 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f5df806d810 0x7f5df806fcd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009630 tx=0x7f5e0803c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:55.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.849+0000 7f5e06ffd640 1 -- 172.21.15.155:0/290036674 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5238+0+0 (secure 0 0 0) 0x7f5e0c0573f0 con 0x7f5e18073310 2024-08-09T14:50:55.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.849+0000 7f5e06ffd640 10 client.4387.objecter ms_dispatch 0x7f5e18000b90 osd_map(28..28 src has 1..28) 2024-08-09T14:50:55.850 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.849+0000 7f5e06ffd640 3 client.4387.objecter handle_osd_map got epochs [28,28] > 0 2024-08-09T14:50:55.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.849+0000 7f5e06ffd640 3 client.4387.objecter handle_osd_map decoding full epoch 28 2024-08-09T14:50:55.851 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:55.849+0000 7f5e06ffd640 20 client.4387.objecter dump_active .. 0 homeless 2024-08-09T14:50:56.086 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.084+0000 7f5e06ffd640 1 -- 172.21.15.155:0/290036674 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7f5e0c059350 con 0x7f5e18073310 2024-08-09T14:50:56.086 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.084+0000 7f5e06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:56.086 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.084+0000 7f5e06ffd640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:56.287 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.286+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/290036674 --> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5e18146b30 con 0x7f5df806d810 2024-08-09T14:50:56.288 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.287+0000 7f5e06ffd640 1 -- 172.21.15.155:0/290036674 <== mgr.4103 v2:172.21.15.155:6800/689799774 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+35819 (secure 0 0 0) 0x7f5e08050c00 con 0x7f5df806d810 2024-08-09T14:50:56.289 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:50:56.289 INFO:teuthology.orchestra.run.smithi155.stderr:dumped all 2024-08-09T14:50:56.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 10 librados: watch_flush enter 2024-08-09T14:50:56.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 10 librados: watch_flush exit 2024-08-09T14:50:56.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 20 client.4387.objecter shutdown clearing up homeless session... 2024-08-09T14:50:56.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 10 client.4387.objecter successfully canceled tick 2024-08-09T14:50:56.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/290036674 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f5df806d810 msgr2=0x7f5df806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:56.291 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 1 --2- 172.21.15.155:0/290036674 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f5df806d810 0x7f5df806fcd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5e08009630 tx=0x7f5e0803c040 comp rx=0 tx=0).stop 2024-08-09T14:50:56.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 10 monclient: shutdown 2024-08-09T14:50:56.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:56.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/290036674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e18073310 msgr2=0x7f5e18138e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:56.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 1 --2- 172.21.15.155:0/290036674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e18073310 0x7f5e18138e20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5e0c00e8a0 tx=0x7f5e0c00ed60 comp rx=0 tx=0).stop 2024-08-09T14:50:56.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/290036674 shutdown_connections 2024-08-09T14:50:56.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 1 --2- 172.21.15.155:0/290036674 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7f5df806d810 0x7f5df806fcd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:56.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 1 --2- 172.21.15.155:0/290036674 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7f5e18143510 0x7f5e18139360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:56.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 1 --2- 172.21.15.155:0/290036674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e18073310 0x7f5e18138e20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:56.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 1 --2- 172.21.15.155:0/290036674 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f5e18067960 0x7f5e18144970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:56.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/290036674 >> 172.21.15.155:0/290036674 conn(0x7f5e180b17a0 msgr2=0x7f5e1806c060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:56.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/290036674 shutdown_connections 2024-08-09T14:50:56.292 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 1 -- 172.21.15.155:0/290036674 wait complete. 2024-08-09T14:50:56.293 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.289+0000 7f5e1c8d6640 1 librados: shutdown 2024-08-09T14:50:56.302 INFO:teuthology.orchestra.run.smithi155.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-08-09T14:50:55.562626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459299,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":115,"num_write_kb":1374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219056,"kb_used_data":2896,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755664,"statfs":{"total":773094113280,"available":772869799936,"internally_reserved":0,"allocated":2965504,"data_stored":1585146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12749,"internal_metadata":221236787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001268"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:47.912680+0000","last_change":"2024-08-09T14:50:47.922102+0000","last_active":"2024-08-09T14:50:47.912680+0000","last_peered":"2024-08-09T14:50:47.912680+0000","last_clean":"2024-08-09T14:50:47.912680+0000","last_became_active":"2024-08-09T14:50:45.816119+0000","last_became_peered":"2024-08-09T14:50:45.816119+0000","last_unstale":"2024-08-09T14:50:47.912680+0000","last_undegraded":"2024-08-09T14:50:47.912680+0000","last_fullsized":"2024-08-09T14:50:47.912680+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-10T19:56:58.452258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.042979287999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:48.067628+0000","last_change":"2024-08-09T14:50:48.104433+0000","last_active":"2024-08-09T14:50:48.067628+0000","last_peered":"2024-08-09T14:50:48.067628+0000","last_clean":"2024-08-09T14:50:48.067628+0000","last_became_active":"2024-08-09T14:50:45.522079+0000","last_became_peered":"2024-08-09T14:50:45.522079+0000","last_unstale":"2024-08-09T14:50:48.067628+0000","last_undegraded":"2024-08-09T14:50:48.067628+0000","last_fullsized":"2024-08-09T14:50:48.067628+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-10T18:50:57.617856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.061146998000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:47.921509+0000","last_change":"2024-08-09T14:50:47.925559+0000","last_active":"2024-08-09T14:50:47.921509+0000","last_peered":"2024-08-09T14:50:47.921509+0000","last_clean":"2024-08-09T14:50:47.921509+0000","last_became_active":"2024-08-09T14:50:44.886965+0000","last_became_peered":"2024-08-09T14:50:44.886965+0000","last_unstale":"2024-08-09T14:50:47.921509+0000","last_undegraded":"2024-08-09T14:50:47.921509+0000","last_fullsized":"2024-08-09T14:50:47.921509+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-11T01:19:47.357851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033223413,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:48.062813+0000","last_change":"2024-08-09T14:50:48.103202+0000","last_active":"2024-08-09T14:50:48.062813+0000","last_peered":"2024-08-09T14:50:48.062813+0000","last_clean":"2024-08-09T14:50:48.062813+0000","last_became_active":"2024-08-09T14:50:45.500606+0000","last_became_peered":"2024-08-09T14:50:45.500606+0000","last_unstale":"2024-08-09T14:50:48.062813+0000","last_undegraded":"2024-08-09T14:50:48.062813+0000","last_fullsized":"2024-08-09T14:50:48.062813+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-11T01:40:33.738679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.072523375000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"27'2","reported_seq":23,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:47.905428+0000","last_change":"2024-08-09T14:50:47.905428+0000","last_active":"2024-08-09T14:50:47.905428+0000","last_peered":"2024-08-09T14:50:47.905428+0000","last_clean":"2024-08-09T14:50:47.905428+0000","last_became_active":"2024-08-09T14:50:45.478451+0000","last_became_peered":"2024-08-09T14:50:45.478451+0000","last_unstale":"2024-08-09T14:50:47.905428+0000","last_undegraded":"2024-08-09T14:50:47.905428+0000","last_fullsized":"2024-08-09T14:50:47.905428+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-11T00:22:14.970158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.058190247,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:48.123636+0000","last_change":"2024-08-09T14:50:48.123636+0000","last_active":"2024-08-09T14:50:48.123636+0000","last_peered":"2024-08-09T14:50:48.123636+0000","last_clean":"2024-08-09T14:50:48.123636+0000","last_became_active":"2024-08-09T14:50:44.778689+0000","last_became_peered":"2024-08-09T14:50:44.778689+0000","last_unstale":"2024-08-09T14:50:48.123636+0000","last_undegraded":"2024-08-09T14:50:48.123636+0000","last_fullsized":"2024-08-09T14:50:48.123636+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-10T21:56:49.407967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.093420954,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:47.796120+0000","last_change":"2024-08-09T14:50:47.838899+0000","last_active":"2024-08-09T14:50:47.796120+0000","last_peered":"2024-08-09T14:50:47.796120+0000","last_clean":"2024-08-09T14:50:47.796120+0000","last_became_active":"2024-08-09T14:50:44.798715+0000","last_became_peered":"2024-08-09T14:50:44.798715+0000","last_unstale":"2024-08-09T14:50:47.796120+0000","last_undegraded":"2024-08-09T14:50:47.796120+0000","last_fullsized":"2024-08-09T14:50:47.796120+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-10T18:48:40.847505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.066653635000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"24'1","reported_seq":22,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-08-09T14:50:47.888484+0000","last_change":"2024-08-09T14:50:47.901627+0000","last_active":"2024-08-09T14:50:47.888484+0000","last_peered":"2024-08-09T14:50:47.888484+0000","last_clean":"2024-08-09T14:50:47.888484+0000","last_became_active":"2024-08-09T14:50:45.496769+0000","last_became_peered":"2024-08-09T14:50:45.496769+0000","last_unstale":"2024-08-09T14:50:47.888484+0000","last_undegraded":"2024-08-09T14:50:47.888484+0000","last_fullsized":"2024-08-09T14:50:47.888484+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:43.214461+0000","last_clean_scrub_stamp":"2024-08-09T14:50:43.214461+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-10T23:34:09.906747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.098838583999999993,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"20'76","reported_seq":135,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-08-09T14:50:54.421600+0000","last_change":"2024-08-09T14:50:32.662579+0000","last_active":"2024-08-09T14:50:54.421600+0000","last_peered":"2024-08-09T14:50:54.421600+0000","last_clean":"2024-08-09T14:50:54.421600+0000","last_became_active":"2024-08-09T14:50:32.629205+0000","last_became_peered":"2024-08-09T14:50:32.629205+0000","last_unstale":"2024-08-09T14:50:54.421600+0000","last_undegraded":"2024-08-09T14:50:54.421600+0000","last_fullsized":"2024-08-09T14:50:54.421600+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-08-09T14:50:30.526481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-09T14:50:30.526481+0000","last_clean_scrub_stamp":"2024-08-09T14:50:30.526481+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-08-10T19:45:32.736889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":21,"seq":90194313228,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":278528,"data_stored":102778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":20,"seq":85899345931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":97312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":19,"seq":81604378636,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":266240,"data_stored":97312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":15,"seq":64424509448,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27684,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344156,"statfs":{"total":96636764160,"available":96608415744,"internally_reserved":0,"allocated":679936,"data_stored":500618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542153,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":270336,"data_stored":97331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27672,"kb_used_data":652,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344168,"statfs":{"total":96636764160,"available":96608428032,"internally_reserved":0,"allocated":667648,"data_stored":495152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542153,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":266240,"data_stored":97312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":270336,"data_stored":97331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-09T14:50:56.303 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-08-09T14:50:56.303 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-09T14:50:56.303 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-08-09T14:50:56.303 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-08-09T14:50:56.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.398+0000 7fd812524640 10 monclient: get_monmap_and_config 2024-08-09T14:50:56.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.398+0000 7fd812524640 10 monclient: build_initial_monmap 2024-08-09T14:50:56.400 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.398+0000 7fd812524640 10 monclient: monmap: 2024-08-09T14:50:56.400 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:56.400 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:56.401 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:50:56.399553+0000 2024-08-09T14:50:56.401 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:50:56.399553+0000 2024-08-09T14:50:56.401 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:56.401 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:56.401 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T14:50:56.401 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T14:50:56.401 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T14:50:56.401 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T14:50:56.401 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T14:50:56.401 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:56.404 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 1 Processor -- start 2024-08-09T14:50:56.404 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 1 -- start start 2024-08-09T14:50:56.404 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 10 monclient: init 2024-08-09T14:50:56.404 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:56.405 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-08-09T14:50:56.405 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd80c06bb80 addr v1:172.21.15.194:6789/0 2024-08-09T14:50:56.405 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd80c073d70 addr v1:172.21.15.155:6789/0 2024-08-09T14:50:56.405 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd80c0b1680 0x7fd80c140130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:56.405 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 10 monclient(hunting): picked mon.noname-a con 0x7fd80c0b1680 addr v2:172.21.15.155:3300/0 2024-08-09T14:50:56.405 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd80c06fef0 con 0x7fd80c073d70 2024-08-09T14:50:56.405 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd80c146090 con 0x7fd80c06bb80 2024-08-09T14:50:56.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:56.406 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7fd80c06bf60 con 0x7fd80c0b1680 2024-08-09T14:50:56.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:56.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.399+0000 7fd812524640 10 monclient(hunting): authenticate will time out at 1263.436949s 2024-08-09T14:50:56.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.400+0000 7fd810ac1640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7fd80c0b1680 0x7fd80c140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:56.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.400+0000 7fd80bfff640 1 --1- >> v1:172.21.15.194:6789/0 conn(0x7fd80c06bb80 0x7fd80c0b07a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.194:6789/0 says I am v1:172.21.15.155:37104/0 (socket says 172.21.15.155:37104) 2024-08-09T14:50:56.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.400+0000 7fd80bfff640 1 -- 172.21.15.155:0/3166632557 learned_addr learned my addr 172.21.15.155:0/3166632557 (peer_addr_for_me v1:172.21.15.155:0/0) 2024-08-09T14:50:56.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.400+0000 7fd810ac1640 10 monclient(hunting): get_auth_request con 0x7fd80c0b1680 auth_method 0 2024-08-09T14:50:56.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.400+0000 7fd810ac1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:56.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.400+0000 7fd810ac1640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:56.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.400+0000 7fd810ac1640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:56.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.402+0000 7fd810ac1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:56.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.402+0000 7fd810ac1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:56.407 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.402+0000 7fd810ac1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:56.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.402+0000 7fd80affd640 1 -- 172.21.15.155:0/3166632557 <== mon.1 v1:172.21.15.194:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 439761961 0 0) 0x7fd80c146090 con 0x7fd80c06bb80 2024-08-09T14:50:56.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.402+0000 7fd80affd640 10 monclient(hunting): _negotiate 2024-08-09T14:50:56.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.402+0000 7fd80affd640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:56.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.402+0000 7fd80affd640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:56.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.402+0000 7fd80affd640 10 monclient(hunting): my global_id is 4390 2024-08-09T14:50:56.408 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.402+0000 7fd80affd640 1 -- 172.21.15.155:0/3166632557 --> v1:172.21.15.194:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7f4003520 con 0x7fd80c06bb80 2024-08-09T14:50:56.409 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.403+0000 7fd810ac1640 10 monclient(hunting): handle_auth_done global_id 4401 payload 274 2024-08-09T14:50:56.409 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.403+0000 7fd810ac1640 1 -- 172.21.15.155:0/3166632557 >> v1:172.21.15.194:6789/0 conn(0x7fd80c06bb80 legacy=0x7fd80c0b07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:56.409 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.403+0000 7fd810ac1640 1 -- 172.21.15.155:0/3166632557 >> v1:172.21.15.155:6789/0 conn(0x7fd80c073d70 legacy=0x7fd80c13fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:56.409 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.403+0000 7fd810ac1640 10 monclient: _finish_hunting 0 2024-08-09T14:50:56.409 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.403+0000 7fd810ac1640 1 monclient: found mon.noname-a 2024-08-09T14:50:56.409 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.403+0000 7fd810ac1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:56.410 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.403+0000 7fd810ac1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.155:3300/0 2024-08-09T14:50:56.410 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.403+0000 7fd810ac1640 1 -- 172.21.15.155:0/3166632557 --> v2:172.21.15.155:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd80c146260 con 0x7fd80c0b1680 2024-08-09T14:50:56.410 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.407+0000 7fd810ac1640 1 --2- 172.21.15.155:0/3166632557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd80c0b1680 0x7fd80c140130 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd804009d90 tx=0x7fd80402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c635ae90c3dbc312 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:56.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd80affd640 1 -- 172.21.15.155:0/3166632557 <== mon.0 v2:172.21.15.155:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd804039070 con 0x7fd80c0b1680 2024-08-09T14:50:56.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd80affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:56.414 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd80affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-08-09T14:50:56.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd80affd640 10 monclient: dump: 2024-08-09T14:50:56.415 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:56.415 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:56.415 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:56.415 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:56.415 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:56.415 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:56.415 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:56.415 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:56.415 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:56.415 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:56.415 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd80affd640 10 monclient: _finish_auth 0 2024-08-09T14:50:56.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd80affd640 10 monclient: _check_auth_tickets 2024-08-09T14:50:56.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd80affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:56.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd80affd640 1 -- 172.21.15.155:0/3166632557 <== mon.0 v2:172.21.15.155:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd80402cd00 con 0x7fd80c0b1680 2024-08-09T14:50:56.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd80affd640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:56.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd80affd640 1 -- 172.21.15.155:0/3166632557 <== mon.0 v2:172.21.15.155:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd804034660 con 0x7fd80c0b1680 2024-08-09T14:50:56.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd80affd640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:56.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd80affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-08-09T14:50:56.416 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd80affd640 10 monclient: dump: 2024-08-09T14:50:56.416 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:56.417 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:56.417 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:56.417 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:56.417 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:56.417 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:56.417 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:56.417 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:56.417 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:56.417 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:56.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd812524640 5 monclient: authenticate success, global_id 4401 2024-08-09T14:50:56.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd812524640 10 monclient: get_monmap_and_config success 2024-08-09T14:50:56.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.412+0000 7fd812524640 4 set_mon_vals no callback set 2024-08-09T14:50:56.417 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.413+0000 7fd812524640 10 monclient: shutdown 2024-08-09T14:50:56.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.413+0000 7fd812524640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:56.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.413+0000 7fd812524640 1 -- 172.21.15.155:0/3166632557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd80c0b1680 msgr2=0x7fd80c140130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:56.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.413+0000 7fd812524640 1 --2- 172.21.15.155:0/3166632557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd80c0b1680 0x7fd80c140130 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd804009d90 tx=0x7fd80402bf50 comp rx=0 tx=0).stop 2024-08-09T14:50:56.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.413+0000 7fd812524640 1 -- 172.21.15.155:0/3166632557 shutdown_connections 2024-08-09T14:50:56.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.413+0000 7fd812524640 1 --2- 172.21.15.155:0/3166632557 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd80c0b1680 0x7fd80c140130 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:56.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.413+0000 7fd812524640 1 -- 172.21.15.155:0/3166632557 >> 172.21.15.155:0/3166632557 conn(0x7fd80c0b5ac0 msgr2=0x7fd80c0b5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:56.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.413+0000 7fd812524640 1 -- 172.21.15.155:0/3166632557 shutdown_connections 2024-08-09T14:50:56.418 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.413+0000 7fd812524640 1 -- 172.21.15.155:0/3166632557 wait complete. 2024-08-09T14:50:56.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.414+0000 7fd812524640 10 monclient: build_initial_monmap 2024-08-09T14:50:56.422 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.414+0000 7fd812524640 10 monclient: monmap: 2024-08-09T14:50:56.422 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T14:50:56.422 INFO:teuthology.orchestra.run.smithi155.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:50:56.422 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 0.000000 2024-08-09T14:50:56.422 INFO:teuthology.orchestra.run.smithi155.stderr:created 0.000000 2024-08-09T14:50:56.423 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T14:50:56.423 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:56.423 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:50:56.423 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:50:56.423 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:50:56.423 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:56.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.414+0000 7fd812524640 1 librados: starting msgr at 2024-08-09T14:50:56.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.414+0000 7fd812524640 1 librados: starting objecter 2024-08-09T14:50:56.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.414+0000 7fd812524640 1 Processor -- start 2024-08-09T14:50:56.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.414+0000 7fd812524640 1 -- start start 2024-08-09T14:50:56.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.414+0000 7fd812524640 1 librados: setting wanted keys 2024-08-09T14:50:56.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.414+0000 7fd812524640 1 librados: calling monclient init 2024-08-09T14:50:56.423 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.414+0000 7fd812524640 10 monclient: init 2024-08-09T14:50:56.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 10 monclient: _reopen_session rank -1 2024-08-09T14:50:56.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 10 monclient: _add_conns ranks=[0,1,2] 2024-08-09T14:50:56.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd80c06bb80 0x7fd80c144dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:56.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 10 monclient(hunting): picked mon.noname-a con 0x7fd80c06bb80 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:50:56.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd80c073d70 0x7fd80c072c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:56.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 10 monclient(hunting): picked mon.noname-b con 0x7fd80c073d70 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:50:56.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd80c0b1680 0x7fd80c142d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:56.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 10 monclient(hunting): picked mon.noname-c con 0x7fd80c0b1680 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:50:56.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:56.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x7fd80c14e640 con 0x7fd80c06bb80 2024-08-09T14:50:56.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:56.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd80c0b0310 con 0x7fd80c073d70 2024-08-09T14:50:56.424 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 10 monclient(hunting): start opening mon connection 2024-08-09T14:50:56.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7fd80c14e7c0 con 0x7fd80c0b1680 2024-08-09T14:50:56.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 10 monclient(hunting): _renew_subs 2024-08-09T14:50:56.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.415+0000 7fd812524640 10 monclient(hunting): authenticate will time out at 1263.452499s 2024-08-09T14:50:56.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80bfff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd80c06bb80 0x7fd80c144dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:56.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd810ac1640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd80c0b1680 0x7fd80c142d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:56.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80b7fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd80c073d70 0x7fd80c072c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:56.425 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80bfff640 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd80c06bb80 0x7fd80c144dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3300/0 says I am v2:172.21.15.155:42344/0 (socket says 172.21.15.155:42344) 2024-08-09T14:50:56.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80bfff640 1 -- 172.21.15.155:0/2515105127 learned_addr learned my addr 172.21.15.155:0/2515105127 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:50:56.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80bfff640 10 monclient(hunting): get_auth_request con 0x7fd80c06bb80 auth_method 0 2024-08-09T14:50:56.426 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:56.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80bfff640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:56.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80bfff640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:56.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80b7fe640 10 monclient(hunting): get_auth_request con 0x7fd80c073d70 auth_method 0 2024-08-09T14:50:56.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:56.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80b7fe640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:56.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:56.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd810ac1640 10 monclient(hunting): get_auth_request con 0x7fd80c0b1680 auth_method 0 2024-08-09T14:50:56.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd810ac1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:50:56.427 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd810ac1640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:50:56.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd810ac1640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:50:56.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:56.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:56.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:56.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:56.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:56.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.416+0000 7fd80bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:56.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.417+0000 7fd810ac1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:50:56.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.417+0000 7fd810ac1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:50:56.428 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.417+0000 7fd810ac1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:50:56.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.419+0000 7fd80b7fe640 10 monclient(hunting): handle_auth_done global_id 4393 payload 824 2024-08-09T14:50:56.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.419+0000 7fd80b7fe640 1 -- 172.21.15.155:0/2515105127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd80c0b1680 msgr2=0x7fd80c142d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:56.429 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.419+0000 7fd80b7fe640 1 --2- 172.21.15.155:0/2515105127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd80c0b1680 0x7fd80c142d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:56.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.419+0000 7fd80b7fe640 1 -- 172.21.15.155:0/2515105127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd80c06bb80 msgr2=0x7fd80c144dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:56.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.419+0000 7fd80b7fe640 1 --2- 172.21.15.155:0/2515105127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd80c06bb80 0x7fd80c144dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:56.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.419+0000 7fd80b7fe640 10 monclient: _finish_hunting 0 2024-08-09T14:50:56.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.419+0000 7fd80b7fe640 1 monclient: found mon.noname-b 2024-08-09T14:50:56.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.419+0000 7fd80b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:50:56.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.419+0000 7fd80b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:50:56.430 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.419+0000 7fd80b7fe640 1 -- 172.21.15.155:0/2515105127 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd80c071b10 con 0x7fd80c073d70 2024-08-09T14:50:56.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.421+0000 7fd80b7fe640 1 --2- 172.21.15.155:0/2515105127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd80c073d70 0x7fd80c072c10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd80000c310 tx=0x7fd80000c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:56.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.421+0000 7fd808ff9640 10 client.?.objecter ms_handle_connect 0x7fd80c073d70 2024-08-09T14:50:56.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.421+0000 7fd808ff9640 10 client.?.objecter resend_mon_ops 2024-08-09T14:50:56.432 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd808ff9640 1 -- 172.21.15.155:0/2515105127 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8000052d0 con 0x7fd80c073d70 2024-08-09T14:50:56.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd808ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:56.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd808ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:50:56.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd808ff9640 10 monclient: dump: 2024-08-09T14:50:56.433 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:56.433 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:56.433 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:56.433 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:56.433 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:56.433 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:56.433 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:56.433 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:56.433 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:56.433 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:56.433 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd808ff9640 10 monclient: _finish_auth 0 2024-08-09T14:50:56.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd808ff9640 10 monclient: _check_auth_tickets 2024-08-09T14:50:56.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd808ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:50:56.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd808ff9640 1 -- 172.21.15.155:0/2515105127 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd800005490 con 0x7fd80c073d70 2024-08-09T14:50:56.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd808ff9640 10 monclient: handle_config config(0 keys) 2024-08-09T14:50:56.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd808ff9640 1 -- 172.21.15.155:0/2515105127 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd800007260 con 0x7fd80c073d70 2024-08-09T14:50:56.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd812524640 5 monclient: authenticate success, global_id 4393 2024-08-09T14:50:56.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd812524640 10 monclient: _renew_subs 2024-08-09T14:50:56.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd812524640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:56.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd812524640 1 -- 172.21.15.155:0/2515105127 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd80c071da0 con 0x7fd80c073d70 2024-08-09T14:50:56.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd808ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:50:56.434 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd808ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:50:56.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd808ff9640 10 monclient: dump: 2024-08-09T14:50:56.435 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 1 2024-08-09T14:50:56.435 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:50:56.435 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:56.435 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:50:56.435 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 19 (squid) 2024-08-09T14:50:56.435 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T14:50:56.435 INFO:teuthology.orchestra.run.smithi155.stderr:0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:50:56.435 INFO:teuthology.orchestra.run.smithi155.stderr:1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:50:56.435 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:50:56.435 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T14:50:56.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd80a7fc640 4 set_mon_vals no callback set 2024-08-09T14:50:56.435 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd812524640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:50:56.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd812524640 10 monclient: _renew_subs 2024-08-09T14:50:56.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd812524640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:56.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd812524640 1 -- 172.21.15.155:0/2515105127 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd80c1434b0 con 0x7fd80c073d70 2024-08-09T14:50:56.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.425+0000 7fd812524640 1 librados: init done 2024-08-09T14:50:56.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.426+0000 7fd812524640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:56.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.426+0000 7fd812524640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:56.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.426+0000 7fd812524640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:56.436 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.426+0000 7fd812524640 1 -- 172.21.15.155:0/2515105127 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd80c0b07a0 con 0x7fd80c073d70 2024-08-09T14:50:56.681 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.680+0000 7fd808ff9640 1 -- 172.21.15.155:0/2515105127 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fd800007400 con 0x7fd80c073d70 2024-08-09T14:50:56.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.680+0000 7fd808ff9640 1 --2- 172.21.15.155:0/2515105127 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fd7f806d810 0x7fd7f806fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:50:56.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.680+0000 7fd80bfff640 1 --2- 172.21.15.155:0/2515105127 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fd7f806d810 0x7fd7f806fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:50:56.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.680+0000 7fd80bfff640 10 monclient: get_auth_request con 0x7fd7f806d810 auth_method 0 2024-08-09T14:50:56.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.681+0000 7fd808ff9640 10 client.4393.objecter ms_handle_connect 0x7fd7f806d810 2024-08-09T14:50:56.682 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.681+0000 7fd80bfff640 1 --2- 172.21.15.155:0/2515105127 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fd7f806d810 0x7fd7f806fcd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc0011e0 tx=0x7fd7fc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:50:56.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.683+0000 7fd808ff9640 1 -- 172.21.15.155:0/2515105127 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5238+0+0 (secure 0 0 0) 0x7fd800056ec0 con 0x7fd80c073d70 2024-08-09T14:50:56.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.683+0000 7fd808ff9640 10 client.4393.objecter ms_dispatch 0x7fd80c000b90 osd_map(28..28 src has 1..28) 2024-08-09T14:50:56.684 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.683+0000 7fd808ff9640 3 client.4393.objecter handle_osd_map got epochs [28,28] > 0 2024-08-09T14:50:56.685 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.683+0000 7fd808ff9640 3 client.4393.objecter handle_osd_map decoding full epoch 28 2024-08-09T14:50:56.685 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.683+0000 7fd808ff9640 20 client.4393.objecter dump_active .. 0 homeless 2024-08-09T14:50:56.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.694+0000 7fd808ff9640 1 -- 172.21.15.155:0/2515105127 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186747 (secure 0 0 0) 0x7fd800058e20 con 0x7fd80c073d70 2024-08-09T14:50:56.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.694+0000 7fd808ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-08-09T14:50:56.695 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.694+0000 7fd808ff9640 10 monclient: _finish_command 1 = system:0 2024-08-09T14:50:56.908 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.907+0000 7fd812524640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-08-09T14:50:56.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.907+0000 7fd812524640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-08-09T14:50:56.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.907+0000 7fd812524640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:50:56.909 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.907+0000 7fd812524640 1 -- 172.21.15.155:0/2515105127 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd80c0b4ba0 con 0x7fd80c073d70 2024-08-09T14:50:56.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.913+0000 7fd808ff9640 1 -- 172.21.15.155:0/2515105127 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd80005d120 con 0x7fd80c073d70 2024-08-09T14:50:56.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.913+0000 7fd808ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-08-09T14:50:56.914 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.913+0000 7fd808ff9640 10 monclient: _finish_command 2 = system:0 2024-08-09T14:50:56.914 INFO:teuthology.orchestra.run.smithi155.stdout: 2024-08-09T14:50:56.914 INFO:teuthology.orchestra.run.smithi155.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-09T14:50:56.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 10 librados: watch_flush enter 2024-08-09T14:50:56.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 10 librados: watch_flush exit 2024-08-09T14:50:56.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 20 client.4393.objecter shutdown clearing up homeless session... 2024-08-09T14:50:56.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 10 client.4393.objecter successfully canceled tick 2024-08-09T14:50:56.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 1 -- 172.21.15.155:0/2515105127 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fd7f806d810 msgr2=0x7fd7f806fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:56.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 1 --2- 172.21.15.155:0/2515105127 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fd7f806d810 0x7fd7f806fcd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd7fc0011e0 tx=0x7fd7fc009040 comp rx=0 tx=0).stop 2024-08-09T14:50:56.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 10 monclient: shutdown 2024-08-09T14:50:56.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:50:56.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 1 -- 172.21.15.155:0/2515105127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd80c073d70 msgr2=0x7fd80c072c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:50:56.917 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 1 --2- 172.21.15.155:0/2515105127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd80c073d70 0x7fd80c072c10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd80000c310 tx=0x7fd80000c7d0 comp rx=0 tx=0).stop 2024-08-09T14:50:56.918 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 1 -- 172.21.15.155:0/2515105127 shutdown_connections 2024-08-09T14:50:56.918 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 1 --2- 172.21.15.155:0/2515105127 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fd7f806d810 0x7fd7f806fcd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:56.918 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 1 --2- 172.21.15.155:0/2515105127 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7fd80c0b1680 0x7fd80c142d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:56.918 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 1 --2- 172.21.15.155:0/2515105127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd80c073d70 0x7fd80c072c10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:56.918 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 1 --2- 172.21.15.155:0/2515105127 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x7fd80c06bb80 0x7fd80c144dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:50:56.918 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 1 -- 172.21.15.155:0/2515105127 >> 172.21.15.155:0/2515105127 conn(0x7fd80c0b5ac0 msgr2=0x7fd80c138a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:50:56.918 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 1 -- 172.21.15.155:0/2515105127 shutdown_connections 2024-08-09T14:50:56.918 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 1 -- 172.21.15.155:0/2515105127 wait complete. 2024-08-09T14:50:56.918 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T14:50:56.915+0000 7fd812524640 1 librados: shutdown 2024-08-09T14:50:56.928 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-08-09T14:50:56.928 INFO:teuthology.run_tasks:Running task workunit... 2024-08-09T14:50:56.942 INFO:tasks.workunit:Pulling workunits from ref 0a88632b45f25ae38493667c7dbd155f6e59b499 2024-08-09T14:50:56.943 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-09T14:50:56.943 DEBUG:teuthology.orchestra.run.smithi155:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-09T14:50:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-09T14:50:56.958 INFO:teuthology.orchestra.run.smithi155.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-09T14:50:56.958 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-09T14:50:57.015 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-09T14:50:57.015 DEBUG:teuthology.orchestra.run.smithi155:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-09T14:50:57.071 INFO:tasks.workunit:timeout=6h 2024-08-09T14:50:57.072 INFO:tasks.workunit:cleanup=True 2024-08-09T14:50:57.072 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 0a88632b45f25ae38493667c7dbd155f6e59b499 2024-08-09T14:50:57.129 INFO:tasks.workunit.client.0.smithi155.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-09T14:50:59.148 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:34.391 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:59.151 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:34.394 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:59.197 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:34.440 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:59.202 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:34.449 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:59.202 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:34.446 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:59.206 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:34.453 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:59.207 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:34.451 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:59.247 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:34.494 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:50:59.254 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:34.498 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:19.154 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:54.396 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:19.201 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:54.448 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:19.205 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:54.452 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:19.209 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:54.453 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:19.222 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:01:54.469 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:19.227 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:01:54.471 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:19.277 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:54.520 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:19.285 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:54.529 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:19.306 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:01:54.550 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:39.220 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:14.463 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:39.273 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:02:14.520 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:39.274 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:02:14.521 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:39.288 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:02:14.535 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:39.306 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:14.549 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:39.325 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:14.568 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:39.389 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:14.633 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:39.445 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:14.635 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:39.445 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:14.659 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:51:46.218 INFO:tasks.workunit.client.0.smithi155.stderr:Note: switching to '0a88632b45f25ae38493667c7dbd155f6e59b499'. 2024-08-09T14:51:46.218 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-08-09T14:51:46.218 INFO:tasks.workunit.client.0.smithi155.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-09T14:51:46.218 INFO:tasks.workunit.client.0.smithi155.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-09T14:51:46.218 INFO:tasks.workunit.client.0.smithi155.stderr:state without impacting any branches by switching back to a branch. 2024-08-09T14:51:46.218 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-08-09T14:51:46.219 INFO:tasks.workunit.client.0.smithi155.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-09T14:51:46.219 INFO:tasks.workunit.client.0.smithi155.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-09T14:51:46.219 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-08-09T14:51:46.219 INFO:tasks.workunit.client.0.smithi155.stderr: git switch -c 2024-08-09T14:51:46.219 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-08-09T14:51:46.219 INFO:tasks.workunit.client.0.smithi155.stderr:Or undo this operation with: 2024-08-09T14:51:46.219 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-08-09T14:51:46.219 INFO:tasks.workunit.client.0.smithi155.stderr: git switch - 2024-08-09T14:51:46.219 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-08-09T14:51:46.219 INFO:tasks.workunit.client.0.smithi155.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-09T14:51:46.219 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-08-09T14:51:46.219 INFO:tasks.workunit.client.0.smithi155.stderr:HEAD is now at 0a88632b45f Merge branch 'wip-67364-squid' of https://github.com/yuvalif/ceph into wip-yuri7-testing-2024-08-08-0804-squid 2024-08-09T14:51:46.223 DEBUG:teuthology.orchestra.run.smithi155:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-09T14:51:46.303 INFO:tasks.workunit.client.0.smithi155.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-08-09T14:51:46.305 INFO:tasks.workunit.client.0.smithi155.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-09T14:51:46.305 INFO:tasks.workunit.client.0.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-08-09T14:51:46.885 INFO:tasks.workunit.client.0.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-08-09T14:51:46.941 INFO:tasks.workunit.client.0.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-08-09T14:51:46.981 INFO:tasks.workunit.client.0.smithi155.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-08-09T14:51:46.982 INFO:tasks.workunit.client.0.smithi155.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-09T14:51:46.982 INFO:tasks.workunit.client.0.smithi155.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-08-09T14:51:47.023 INFO:tasks.workunit.client.0.smithi155.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-08-09T14:51:47.026 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-08-09T14:51:47.026 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-09T14:51:47.082 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-08-09T14:51:47.083 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-08-09T14:51:47.083 DEBUG:teuthology.orchestra.run.smithi155:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a88632b45f25ae38493667c7dbd155f6e59b499 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-08-09T14:51:47.142 INFO:tasks.workunit.client.0.smithi155.stderr:+ parallel=1 2024-08-09T14:51:47.142 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' '' = --serial ']' 2024-08-09T14:51:47.142 INFO:tasks.workunit.client.0.smithi155.stderr:+ crimson=0 2024-08-09T14:51:47.142 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' '' = --crimson ']' 2024-08-09T14:51:47.142 INFO:tasks.workunit.client.0.smithi155.stderr:+ color= 2024-08-09T14:51:47.143 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -t 1 ']' 2024-08-09T14:51:47.143 INFO:tasks.workunit.client.0.smithi155.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-08-09T14:51:47.143 INFO:tasks.workunit.client.0.smithi155.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-09T14:51:47.143 INFO:tasks.workunit.client.0.smithi155.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-08-09T14:51:47.144 INFO:tasks.workunit.client.0.smithi155.stderr:+ declare -A pids 2024-08-09T14:51:47.144 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.144 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.144 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_aio 2024-08-09T14:51:47.144 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_aio' 2024-08-09T14:51:47.144 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_aio 2024-08-09T14:51:47.144 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.145 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_aio 2024-08-09T14:51:47.145 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41392 2024-08-09T14:51:47.145 INFO:tasks.workunit.client.0.smithi155.stdout:test api_aio on pid 41392 2024-08-09T14:51:47.148 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_aio on pid 41392' 2024-08-09T14:51:47.148 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41392 2024-08-09T14:51:47.148 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.148 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.148 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-08-09T14:51:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_aio_pp 2024-08-09T14:51:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_aio_pp' 2024-08-09T14:51:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_aio_pp 2024-08-09T14:51:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.150 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.150 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.150 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.150 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.150 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.150 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.150 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.150 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.150 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.150 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.150 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.150 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.150 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.151 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.151 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.151 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.151 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_aio_pp 2024-08-09T14:51:47.151 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41399 2024-08-09T14:51:47.151 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_aio_pp on pid 41399' 2024-08-09T14:51:47.151 INFO:tasks.workunit.client.0.smithi155.stdout:test api_aio_pp on pid 41399 2024-08-09T14:51:47.154 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41399 2024-08-09T14:51:47.154 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.154 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.155 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-08-09T14:51:47.155 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_io 2024-08-09T14:51:47.155 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_io' 2024-08-09T14:51:47.155 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_io 2024-08-09T14:51:47.155 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.155 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.155 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.155 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.155 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.155 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.155 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.155 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.155 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.156 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.156 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.156 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.156 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.156 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.156 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.156 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.156 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.156 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.156 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.156 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.156 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.156 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.157 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.157 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.157 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.157 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.157 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.157 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.157 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.157 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.157 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.157 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.157 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.157 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.157 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.157 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.158 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.158 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.158 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.158 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.158 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.158 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.158 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.158 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.158 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.158 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.158 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_io 2024-08-09T14:51:47.158 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41408 2024-08-09T14:51:47.159 INFO:tasks.workunit.client.0.smithi155.stdout:test api_io on pid 41408 2024-08-09T14:51:47.162 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_io on pid 41408' 2024-08-09T14:51:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41408 2024-08-09T14:51:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-08-09T14:51:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_io_pp 2024-08-09T14:51:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_io_pp' 2024-08-09T14:51:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_io_pp 2024-08-09T14:51:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.164 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.164 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.164 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.164 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.164 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.164 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.164 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.164 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.164 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.164 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.164 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.164 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.164 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.164 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.165 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.165 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.165 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.165 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.165 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.165 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.165 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.165 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.165 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.165 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.165 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.165 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.165 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.166 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.166 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.166 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.166 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.166 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.166 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.166 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.166 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.166 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.166 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.166 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.166 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.166 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.167 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.167 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.167 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.167 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.167 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.167 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.167 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.167 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.167 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.167 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.167 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_io_pp 2024-08-09T14:51:47.167 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41419 2024-08-09T14:51:47.167 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_io_pp on pid 41419' 2024-08-09T14:51:47.168 INFO:tasks.workunit.client.0.smithi155.stdout:test api_io_pp on pid 41419 2024-08-09T14:51:47.173 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41419 2024-08-09T14:51:47.173 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.173 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.173 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-08-09T14:51:47.173 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_asio 2024-08-09T14:51:47.173 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_asio' 2024-08-09T14:51:47.173 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_asio 2024-08-09T14:51:47.173 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.173 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.174 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.174 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.174 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.174 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.174 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.174 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.174 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.174 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.174 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.174 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.174 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.174 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.174 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.175 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.175 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.175 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.175 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.175 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.175 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.175 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.175 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.175 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.175 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.175 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.175 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.175 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.175 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.176 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.176 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.176 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.176 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.176 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.176 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.176 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.176 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.176 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.176 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.176 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.176 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.176 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.177 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.177 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.177 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.177 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.177 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.177 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.177 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.177 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.177 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.177 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.177 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.177 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.177 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.177 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.178 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.178 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.178 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.178 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.178 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.178 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.178 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.178 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.178 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.178 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.178 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.178 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.178 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.178 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.179 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.179 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.179 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.179 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_asio 2024-08-09T14:51:47.179 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41433 2024-08-09T14:51:47.179 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_asio on pid 41433' 2024-08-09T14:51:47.179 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-08-09T14:51:47.179 INFO:tasks.workunit.client.0.smithi155.stdout:test api_asio on pid 41433 2024-08-09T14:51:47.186 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41433 2024-08-09T14:51:47.186 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.186 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.186 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_list 2024-08-09T14:51:47.186 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_list' 2024-08-09T14:51:47.186 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.187 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.187 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.187 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.187 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.187 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_list 2024-08-09T14:51:47.187 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.187 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.187 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.187 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.187 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.187 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.187 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.187 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.187 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.188 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.188 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.188 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.188 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.188 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.188 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.188 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.188 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.188 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.188 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.188 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.188 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.188 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.189 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.189 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.189 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.189 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.189 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.189 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.189 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.189 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.189 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.189 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.189 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.189 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.189 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.190 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.190 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.190 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.190 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.190 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.190 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.190 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.190 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.190 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.190 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.190 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.190 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.190 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.190 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.191 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.191 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.191 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.191 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.191 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.191 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.191 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.191 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.191 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.191 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.191 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.191 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.191 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.192 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.192 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.192 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.192 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.192 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.192 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.192 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.192 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.192 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.192 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.192 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.192 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.192 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.193 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.193 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.193 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.193 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.193 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.193 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.193 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.193 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.193 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.193 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.193 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.193 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.193 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.194 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.194 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.194 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.194 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.194 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.194 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.194 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.194 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_list 2024-08-09T14:51:47.194 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41450 2024-08-09T14:51:47.194 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.194 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41392/exe 2024-08-09T14:51:47.194 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.194 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-08-09T14:51:47.194 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.195 INFO:tasks.workunit.client.0.smithi155.stdout:test api_list on pid 41450 2024-08-09T14:51:47.201 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_list on pid 41450' 2024-08-09T14:51:47.201 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41450 2024-08-09T14:51:47.201 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.201 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.201 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.201 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.201 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_lock 2024-08-09T14:51:47.201 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.202 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_lock' 2024-08-09T14:51:47.202 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.202 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_lock 2024-08-09T14:51:47.202 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.202 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.202 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.202 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.202 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.202 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.202 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.202 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.202 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.202 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.203 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.203 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.203 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.203 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.203 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.203 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.203 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.203 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.203 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.203 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.203 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.203 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.203 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.204 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.204 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.204 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.204 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.204 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.204 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.204 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.204 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.204 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.204 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.204 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.204 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.204 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.205 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.205 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.205 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.205 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.205 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.205 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.205 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.205 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.205 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.205 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.205 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.205 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.205 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.205 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.206 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.206 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.206 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.206 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.206 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.206 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.206 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.206 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.206 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.206 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-08-09T14:51:47.206 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.206 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.206 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.207 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.207 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.207 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.207 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.207 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.207 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.207 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.207 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.207 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.207 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.207 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.207 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.207 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_aio: /' 2024-08-09T14:51:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_aio.log 2024-08-09T14:51:47.209 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.209 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41399/exe 2024-08-09T14:51:47.209 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.209 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.209 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.209 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.209 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_lock 2024-08-09T14:51:47.209 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.209 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.209 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.209 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.209 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.209 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.209 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.210 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.210 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.210 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.210 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.210 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.210 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41469 2024-08-09T14:51:47.210 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_lock on pid 41469' 2024-08-09T14:51:47.210 INFO:tasks.workunit.client.0.smithi155.stdout:test api_lock on pid 41469 2024-08-09T14:51:47.217 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41469 2024-08-09T14:51:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_lock_pp 2024-08-09T14:51:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_lock_pp' 2024-08-09T14:51:47.218 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-08-09T14:51:47.219 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.219 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.219 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.219 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.219 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.219 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_lock_pp 2024-08-09T14:51:47.219 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.219 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.219 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.219 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.219 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.219 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.219 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.220 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.220 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.220 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.220 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.220 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.220 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.220 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.220 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.220 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.220 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.220 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.220 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.220 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.222 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.222 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.222 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.222 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.222 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.222 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.222 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.222 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-08-09T14:51:47.222 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.222 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.222 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.222 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.223 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.223 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.223 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.223 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.223 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.223 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.223 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.223 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.223 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.223 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.223 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_aio_pp: /' 2024-08-09T14:51:47.223 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.223 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.223 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.224 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.225 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.225 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.225 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.225 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.225 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.225 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.225 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_lock_pp 2024-08-09T14:51:47.225 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41488 2024-08-09T14:51:47.225 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_lock_pp on pid 41488' 2024-08-09T14:51:47.225 INFO:tasks.workunit.client.0.smithi155.stdout:test api_lock_pp on pid 41488 2024-08-09T14:51:47.234 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.234 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41488 2024-08-09T14:51:47.234 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.235 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.235 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_misc 2024-08-09T14:51:47.235 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_misc' 2024-08-09T14:51:47.235 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-08-09T14:51:47.235 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41408/exe 2024-08-09T14:51:47.235 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.235 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.235 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.235 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.235 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.235 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.235 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.235 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_misc 2024-08-09T14:51:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.240 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.240 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.240 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.240 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.240 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.240 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.240 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.240 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.240 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.240 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.240 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.240 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.240 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.240 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.242 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.242 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.242 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.242 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.242 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.242 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.242 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.242 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.242 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-08-09T14:51:47.242 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.242 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.242 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.242 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.243 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.243 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.243 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.243 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.243 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.243 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.243 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.243 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.243 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.243 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.243 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.243 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.243 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.243 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.244 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.244 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.244 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.244 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.244 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.244 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.244 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_io.log 2024-08-09T14:51:47.244 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.244 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41419/exe 2024-08-09T14:51:47.244 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_misc 2024-08-09T14:51:47.244 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41510 2024-08-09T14:51:47.244 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_misc on pid 41510' 2024-08-09T14:51:47.244 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.245 INFO:tasks.workunit.client.0.smithi155.stdout:test api_misc on pid 41510 2024-08-09T14:51:47.254 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.254 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.254 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41510 2024-08-09T14:51:47.254 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.254 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-08-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.255 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.256 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_io: /' 2024-08-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.257 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_misc_pp 2024-08-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.258 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.259 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.259 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.259 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_misc_pp' 2024-08-09T14:51:47.259 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_io_pp: /' 2024-08-09T14:51:47.259 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.259 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.259 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-08-09T14:51:47.259 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.259 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.259 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.259 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.259 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.259 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.260 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.260 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.260 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.260 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.260 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.260 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.260 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.260 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.260 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.260 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.260 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.260 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.260 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.260 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.261 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.261 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.261 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.261 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.261 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.261 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.261 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.261 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.261 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.261 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.261 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.261 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.262 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.262 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.262 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.262 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.262 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.262 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.262 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.262 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.262 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.262 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.262 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.262 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.262 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-08-09T14:51:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.263 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.264 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.264 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_misc_pp 2024-08-09T14:51:47.264 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.264 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.264 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.264 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.264 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.264 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.264 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.264 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_misc_pp 2024-08-09T14:51:47.264 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41433/exe 2024-08-09T14:51:47.264 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.264 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41530 2024-08-09T14:51:47.264 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_misc_pp on pid 41530' 2024-08-09T14:51:47.265 INFO:tasks.workunit.client.0.smithi155.stdout:test api_misc_pp on pid 41530 2024-08-09T14:51:47.268 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41530 2024-08-09T14:51:47.268 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.268 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.268 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-08-09T14:51:47.268 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_tier_pp 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_tier_pp' 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_tier_pp 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.269 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.270 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.270 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.270 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.270 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.270 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.270 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.270 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.270 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.270 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.270 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.270 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.270 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-08-09T14:51:47.270 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.270 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.271 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.271 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.271 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.271 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.271 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.271 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.271 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.271 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_tier_pp 2024-08-09T14:51:47.271 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41543 2024-08-09T14:51:47.271 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_tier_pp on pid 41543' 2024-08-09T14:51:47.271 INFO:tasks.workunit.client.0.smithi155.stdout:test api_tier_pp on pid 41543 2024-08-09T14:51:47.277 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41543 2024-08-09T14:51:47.277 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.277 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-08-09T14:51:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_pool 2024-08-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.279 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.280 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_asio.log 2024-08-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_pool' 2024-08-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_pool 2024-08-09T14:51:47.281 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.282 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.282 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.282 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.282 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.282 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.282 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.282 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.282 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.282 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.282 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.282 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.282 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.282 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.282 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.283 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.283 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.283 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.283 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.283 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.283 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_pool 2024-08-09T14:51:47.283 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.283 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.283 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41554 2024-08-09T14:51:47.283 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_pool on pid 41554' 2024-08-09T14:51:47.283 INFO:tasks.workunit.client.0.smithi155.stdout:test api_pool on pid 41554 2024-08-09T14:51:47.310 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41554 2024-08-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-08-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.311 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.312 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_asio: /' 2024-08-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.313 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.314 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.314 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.314 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.314 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.314 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.314 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.314 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.314 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.314 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.314 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.314 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.314 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.314 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.315 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.316 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.316 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.316 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.316 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.316 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.316 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.316 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.316 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.316 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.316 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.316 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.316 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.316 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.317 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.317 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.317 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.317 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.317 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.317 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.317 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.317 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.317 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.317 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.317 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.317 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.318 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.318 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.318 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.318 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.318 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.318 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.318 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.318 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.318 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.318 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.318 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.318 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.318 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.319 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.319 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.319 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.319 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.319 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41450/exe 2024-08-09T14:51:47.319 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.319 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.319 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.319 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.319 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.319 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.319 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.319 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.320 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.320 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.320 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.320 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.320 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.320 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.320 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.320 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.320 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.320 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.320 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.320 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.320 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.321 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.321 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.321 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.321 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.321 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.321 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.321 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.321 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.321 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.321 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.321 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.321 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.322 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.322 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.322 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.322 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.322 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.322 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.322 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.322 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.322 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.322 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.322 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.322 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.322 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.323 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.324 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.324 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.324 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.324 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.324 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.324 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.324 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.324 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.324 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.324 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.324 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.324 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.324 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.324 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.325 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.326 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-08-09T14:51:47.326 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.326 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.326 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.326 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.326 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.326 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.326 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.326 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.326 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.326 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.326 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.326 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_list.log 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_list: /' 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_snapshots 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.327 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_snapshots' 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.328 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.329 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.330 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.330 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.330 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.330 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_snapshots 2024-08-09T14:51:47.330 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.330 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41510/exe 2024-08-09T14:51:47.330 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.330 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.330 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.330 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.330 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.330 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.330 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.331 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.331 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.331 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.331 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.331 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.331 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.331 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.331 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.331 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.331 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41488/exe 2024-08-09T14:51:47.331 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.331 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.331 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_misc.log 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.332 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.333 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.334 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.334 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.334 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.334 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.334 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.334 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.334 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.334 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.334 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.334 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.334 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.334 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.334 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.335 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.336 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.337 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.337 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.337 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.337 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.337 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.337 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_snapshots 2024-08-09T14:51:47.337 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.337 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41624 2024-08-09T14:51:47.337 INFO:tasks.workunit.client.0.smithi155.stdout:test api_snapshots on pid 41624 2024-08-09T14:51:47.346 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_snapshots on pid 41624' 2024-08-09T14:51:47.346 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41624 2024-08-09T14:51:47.346 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.346 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.346 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.346 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_snapshots_pp 2024-08-09T14:51:47.346 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-08-09T14:51:47.346 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_snapshots_pp' 2024-08-09T14:51:47.346 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.346 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.346 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.346 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.347 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.347 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.347 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.347 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.347 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-08-09T14:51:47.347 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.347 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.347 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.347 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.347 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.347 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.347 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.347 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.348 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.348 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.348 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_misc: /' 2024-08-09T14:51:47.348 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.348 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.348 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.348 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.348 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.348 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.348 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.348 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.348 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.349 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.349 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.349 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_lock_pp: /' 2024-08-09T14:51:47.349 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.349 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.349 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.349 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.349 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.349 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.349 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.349 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.349 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.349 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-08-09T14:51:47.350 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.350 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.350 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.350 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.350 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.350 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.350 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.350 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.350 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.350 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.350 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.350 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.352 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.352 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.352 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.352 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.352 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.352 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.352 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.352 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.352 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.352 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.352 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.352 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_snapshots_pp 2024-08-09T14:51:47.352 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.353 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.354 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_snapshots_pp 2024-08-09T14:51:47.354 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.354 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.354 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41657 2024-08-09T14:51:47.354 INFO:tasks.workunit.client.0.smithi155.stdout:test api_snapshots_pp on pid 41657 2024-08-09T14:51:47.369 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_snapshots_pp on pid 41657' 2024-08-09T14:51:47.369 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41657 2024-08-09T14:51:47.369 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.369 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.369 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-08-09T14:51:47.369 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.370 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.370 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.370 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_stat 2024-08-09T14:51:47.370 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_stat' 2024-08-09T14:51:47.370 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.370 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41469/exe 2024-08-09T14:51:47.370 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_stat 2024-08-09T14:51:47.370 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.370 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.370 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.370 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.370 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.370 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.371 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.371 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.371 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.371 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.371 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.371 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.371 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.371 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.371 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.371 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.371 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.371 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.371 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.372 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.372 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.372 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-08-09T14:51:47.372 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_lock: /' 2024-08-09T14:51:47.372 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_lock.log 2024-08-09T14:51:47.372 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.372 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.372 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.372 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.372 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.372 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.372 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.372 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.372 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.373 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.373 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.373 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.373 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.373 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.373 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.373 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.373 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.373 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.373 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.373 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.373 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.374 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.374 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.374 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.374 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.374 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.374 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.374 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.374 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.374 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.374 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.374 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.374 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.374 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.375 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.375 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.375 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.375 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.375 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.375 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41554/exe 2024-08-09T14:51:47.375 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.375 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.375 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.375 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.375 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.375 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.375 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.375 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.376 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.376 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.376 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.376 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.376 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.376 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.376 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.376 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.376 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_pool.log 2024-08-09T14:51:47.376 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.376 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.376 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.376 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.377 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.377 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.377 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.377 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.377 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.377 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.377 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.377 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.377 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.377 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.377 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.377 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.377 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.378 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.378 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.378 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.378 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_pool: /' 2024-08-09T14:51:47.378 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.378 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.378 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.378 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.378 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.378 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.378 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-08-09T14:51:47.378 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.378 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.380 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.380 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.380 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.380 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41543/exe 2024-08-09T14:51:47.380 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.380 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41530/exe 2024-08-09T14:51:47.380 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.380 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.380 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.380 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.380 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.380 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.380 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.381 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.383 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.383 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.383 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.383 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.383 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.383 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.383 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.383 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.383 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.383 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.383 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_tier_pp: /' 2024-08-09T14:51:47.383 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.383 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.384 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.384 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_misc_pp: /' 2024-08-09T14:51:47.384 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_stat 2024-08-09T14:51:47.384 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41713 2024-08-09T14:51:47.384 INFO:tasks.workunit.client.0.smithi155.stdout:test api_stat on pid 41713 2024-08-09T14:51:47.385 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_stat on pid 41713' 2024-08-09T14:51:47.385 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41713 2024-08-09T14:51:47.385 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.385 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.385 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-08-09T14:51:47.386 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_stat_pp 2024-08-09T14:51:47.386 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_stat_pp' 2024-08-09T14:51:47.386 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.386 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-08-09T14:51:47.386 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.386 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.386 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.386 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_stat_pp 2024-08-09T14:51:47.386 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-08-09T14:51:47.386 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [----------] Global test environment set-up. 2024-08-09T14:51:47.386 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [----------] 12 tests from AsioRados 2024-08-09T14:51:47.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.261+0000 7fc3cb6c6780 10 monclient: get_monmap_and_config 2024-08-09T14:51:47.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.261+0000 7fc3cb6c6780 10 monclient: build_initial_monmap 2024-08-09T14:51:47.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.261+0000 7fc3cb6c6780 10 monclient: monmap: 2024-08-09T14:51:47.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: epoch 0 2024-08-09T14:51:47.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:51:47.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: last_changed 0.000000 2024-08-09T14:51:47.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: created 0.000000 2024-08-09T14:51:47.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: min_mon_release 0 (unknown) 2024-08-09T14:51:47.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: election_strategy: 1 2024-08-09T14:51:47.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:51:47.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:51:47.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:51:47.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.262+0000 7fc3cb6c6780 1 Processor -- start 2024-08-09T14:51:47.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.262+0000 7fc3cb6c6780 1 -- start start 2024-08-09T14:51:47.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.262+0000 7fc3cb6c6780 10 monclient: init 2024-08-09T14:51:47.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 10 monclient: _reopen_session rank -1 2024-08-09T14:51:47.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 10 monclient: _add_conns ranks=[0,2,1] 2024-08-09T14:51:47.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a86d4642e0 0x55a86d461ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:51:47.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 10 monclient(hunting): picked mon.noname-a con 0x55a86d4642e0 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:51:47.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a86d4f63d0 0x55a86d4f8850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:51:47.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 10 monclient(hunting): picked mon.noname-c con 0x55a86d4f63d0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:51:47.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a86d461130 0x55a86d45fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:51:47.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 10 monclient(hunting): picked mon.noname-b con 0x55a86d461130 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:51:47.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 10 monclient(hunting): start opening mon connection 2024-08-09T14:51:47.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a86d383c00 con 0x55a86d4642e0 2024-08-09T14:51:47.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 10 monclient(hunting): start opening mon connection 2024-08-09T14:51:47.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a86d3d8e70 con 0x55a86d461130 2024-08-09T14:51:47.389 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 10 monclient(hunting): start opening mon connection 2024-08-09T14:51:47.389 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x55a86d462900 con 0x55a86d4f63d0 2024-08-09T14:51:47.389 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 10 monclient(hunting): _renew_subs 2024-08-09T14:51:47.389 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.263+0000 7fc3cb6c6780 10 monclient(hunting): authenticate will time out at 1314.300280s 2024-08-09T14:51:47.389 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-08-09T14:51:47.389 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-08-09T14:51:47.389 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.264+0000 7fc3c89c1640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a86d4f63d0 0x55a86d4f8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:51:47.390 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.264+0000 7fc3c89c1640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a86d4f63d0 0x55a86d4f8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.155:3301/0 says I am v2:172.21.15.155:57606/0 (socket says 172.21.15.155:57606) 2024-08-09T14:51:47.390 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.264+0000 7fc3c89c1640 1 -- 172.21.15.155:0/2531537878 learned_addr learned my addr 172.21.15.155:0/2531537878 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:51:47.390 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.264+0000 7fc3c89c1640 10 monclient(hunting): get_auth_request con 0x55a86d4f63d0 auth_method 0 2024-08-09T14:51:47.390 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.264+0000 7fc3c89c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:51:47.390 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.264+0000 7fc3c89c1640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:51:47.390 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.264+0000 7fc3c89c1640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:51:47.390 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.264+0000 7fc3c91c2640 1 --2- 172.21.15.155:0/2531537878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a86d4642e0 0x55a86d461ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:51:47.390 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.265+0000 7fc3c91c2640 10 monclient(hunting): get_auth_request con 0x55a86d4642e0 auth_method 0 2024-08-09T14:51:47.390 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.265+0000 7fc3c91c2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:51:47.390 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.265+0000 7fc3c91c2640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:51:47.390 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.265+0000 7fc3c91c2640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:51:47.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.265+0000 7fc3c89c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:51:47.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.265+0000 7fc3c89c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:51:47.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.265+0000 7fc3c89c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:51:47.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.265+0000 7fc3c91c2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:51:47.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.265+0000 7fc3c91c2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:51:47.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.265+0000 7fc3c91c2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:51:47.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.266+0000 7fc3c99c3640 1 --2- 172.21.15.155:0/2531537878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a86d461130 0x55a86d45fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:51:47.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.267+0000 7fc3c89c1640 10 monclient(hunting): handle_auth_done global_id 4364 payload 274 2024-08-09T14:51:47.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.267+0000 7fc3c89c1640 1 -- 172.21.15.155:0/2531537878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a86d461130 msgr2=0x55a86d45fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:51:47.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.267+0000 7fc3c89c1640 1 --2- 172.21.15.155:0/2531537878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a86d461130 0x55a86d45fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:51:47.392 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_stat_pp 2024-08-09T14:51:47.392 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41740 2024-08-09T14:51:47.392 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_stat_pp on pid 41740' 2024-08-09T14:51:47.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.267+0000 7fc3c89c1640 1 -- 172.21.15.155:0/253153787test api_stat_pp on pid 41740 2024-08-09T14:51:47.411 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41740 2024-08-09T14:51:47.411 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.412 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.412 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-08-09T14:51:47.412 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.412 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.412 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.412 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.412 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.412 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.412 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.412 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.412 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.412 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.413 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.413 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.413 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.413 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.413 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.413 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.413 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.413 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.413 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.413 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.413 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.413 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.413 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.414 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.414 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.414 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.414 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-08-09T14:51:47.414 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.414 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.414 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.414 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.414 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.414 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.414 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.414 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.415 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.415 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.415 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.415 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.415 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.415 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.415 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.415 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.415 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.415 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.415 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.415 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.415 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.416 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.416 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.416 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.416 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.416 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.416 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.416 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.416 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.416 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.416 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.416 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.416 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.416 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.416 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.417 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.417 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.417 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.417 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.417 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.417 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.417 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.417 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.417 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.417 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.417 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.417 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.417 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.418 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.418 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.418 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.418 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.418 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.418 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.418 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.418 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.418 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.418 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.418 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.418 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.418 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.419 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.419 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.419 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.419 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41624/exe 2024-08-09T14:51:47.419 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.419 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.419 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.419 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.419 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.419 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.419 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.419 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.419 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.419 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.420 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.420 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_snapshots: /' 2024-08-09T14:51:47.420 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.420 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.420 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.420 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.420 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.420 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.420 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.420 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.420 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.420 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.421 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.422 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.422 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.422 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.422 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.422 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.422 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-08-09T14:51:47.422 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_watch_notify 2024-08-09T14:51:47.422 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.422 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.422 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.422 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_watch_notify' 2024-08-09T14:51:47.422 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.422 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.423 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.423 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.423 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.423 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.423 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-08-09T14:51:47.423 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.423 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.423 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.423 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.423 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.423 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.423 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.423 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.424 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.424 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.424 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.424 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.424 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.424 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.424 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.424 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.424 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.424 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.424 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.424 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.424 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.425 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.425 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.425 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_watch_notify 2024-08-09T14:51:47.425 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.425 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.425 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.425 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.425 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.425 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.425 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.425 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.425 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.425 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.426 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.426 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.426 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.426 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.426 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.426 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.426 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.426 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.426 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.426 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.426 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.426 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.426 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.427 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.429 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.430 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.430 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.430 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.430 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_watch_notify 2024-08-09T14:51:47.430 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.430 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.430 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.430 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.430 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.430 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.430 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41811 2024-08-09T14:51:47.431 INFO:tasks.workunit.client.0.smithi155.stdout:test api_watch_notify on pid 41811 2024-08-09T14:51:47.437 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_watch_notify on pid 41811' 2024-08-09T14:51:47.437 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41811 2024-08-09T14:51:47.437 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.437 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.437 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-08-09T14:51:47.437 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_watch_notify_pp 2024-08-09T14:51:47.437 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_watch_notify_pp' 2024-08-09T14:51:47.437 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_watch_notify_pp 2024-08-09T14:51:47.437 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.438 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.438 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.438 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.438 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.438 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.438 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.438 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.438 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.438 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41740/exe 2024-08-09T14:51:47.438 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.438 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.438 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.439 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.439 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.439 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.439 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.439 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.439 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.439 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.439 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.439 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.439 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.439 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.439 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.439 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.442 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.442 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.442 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.442 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.442 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.442 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.442 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.442 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.442 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.442 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.442 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.442 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.442 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-08-09T14:51:47.443 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_watch_notify_pp 2024-08-09T14:51:47.443 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-08-09T14:51:47.443 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41828 2024-08-09T14:51:47.443 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_watch_notify_pp on pid 41828' 2024-08-09T14:51:47.443 INFO:tasks.workunit.client.0.smithi155.stdout:test api_watch_notify_pp on pid 41828 2024-08-09T14:51:47.452 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41828 2024-08-09T14:51:47.452 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.452 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.452 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.452 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.452 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.452 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_stat_pp: /' 2024-08-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.453 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-08-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41713/exe 2024-08-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.454 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.455 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.455 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.455 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.455 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.455 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.455 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.455 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.455 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.455 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.455 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.455 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.455 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.455 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.456 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.456 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_cmd 2024-08-09T14:51:47.456 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.456 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.456 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_cmd' 2024-08-09T14:51:47.456 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.456 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.456 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.456 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.456 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.456 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_cmd 2024-08-09T14:51:47.456 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.456 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.456 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.457 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.457 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.457 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.457 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.457 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.457 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.457 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.457 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.457 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.457 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.457 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.457 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.457 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.458 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.458 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.458 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41657/exe 2024-08-09T14:51:47.458 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.458 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.458 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.458 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.458 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.458 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.458 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.458 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.458 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.459 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.459 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.459 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.459 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.459 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.459 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.459 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.459 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.459 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.459 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.459 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.459 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.459 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.459 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.460 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.460 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.460 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.460 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.460 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_cmd 2024-08-09T14:51:47.460 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.460 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.460 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.460 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41844 2024-08-09T14:51:47.460 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_cmd on pid 41844' 2024-08-09T14:51:47.460 INFO:tasks.workunit.client.0.smithi155.stdout:test api_cmd on pid 41844 2024-08-09T14:51:47.464 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41844 2024-08-09T14:51:47.464 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.464 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.464 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.464 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.464 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-08-09T14:51:47.464 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.464 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.464 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.465 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.465 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.465 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.465 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.465 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.465 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.465 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.465 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.465 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.465 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.465 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.465 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.465 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.466 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-08-09T14:51:47.466 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-08-09T14:51:47.466 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.466 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.466 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.466 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.466 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.466 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.466 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.466 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.466 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.466 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-08-09T14:51:47.466 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-08-09T14:51:47.467 INFO:tasks.workunit.client.0.smithi155.stdout:8 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a86d4642e0 msgr2=0x55a86d461ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-08-09T14:51:47.467 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.267+0000 7fc3c89c1640 1 --2- 172.21.15.155:0/2531537878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a86d4642e0 0x55a86d461ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:51:47.467 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.267+0000 7fc3c89c1640 10 monclient: _finish_hunting 0 2024-08-09T14:51:47.467 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.267+0000 7fc3c89c1640 1 monclient: found mon.noname-c 2024-08-09T14:51:47.467 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.267+0000 7fc3c89c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:51:47.467 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.267+0000 7fc3c89c1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.155:3301/0 2024-08-09T14:51:47.467 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.267+0000 7fc3c89c1640 1 -- 172.21.15.155:0/2531537878 --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86d35dca0 con 0x55a86d4f63d0 2024-08-09T14:51:47.467 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.276+0000 7fc3c89c1640 1 --2- 172.21.15.155:0/2531537878 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a86d4f63d0 0x55a86d4f8850 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc001e00 tx=0x7fc3bc00b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b78a3a18c9c4532 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:51:47.467 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3bb7fe640 1 -- 172.21.15.155:0/2531537878 <== mon.2 v2:172.21.15.155:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc00be40 con 0x55a86d4f63d0 2024-08-09T14:51:47.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:51:47.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3bb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-08-09T14:51:47.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3bb7fe640 10 monclient: dump: 2024-08-09T14:51:47.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: epoch 1 2024-08-09T14:51:47.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:51:47.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:51:47.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:51:47.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: min_mon_release 19 (squid) 2024-08-09T14:51:47.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: election_strategy: 1 2024-08-09T14:51:47.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:51:47.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:51:47.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:51:47.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3bb7fe640 10 monclient: _finish_auth 0 2024-08-09T14:51:47.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3bb7fe640 10 monclient: _check_auth_tickets 2024-08-09T14:51:47.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:51:47.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3bb7fe640 1 -- 172.21.15.155:0/2531537878 <== mon.2 v2:172.21.15.155:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3bc00f400 con 0x55a86d4f63d0 2024-08-09T14:51:47.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3bb7fe640 10 monclient: handle_config config(0 keys) 2024-08-09T14:51:47.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3bb7fe640 1 -- 172.21.15.155:0/2531537878 <== mon.2 v2:172.21.15.155:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc00f7d0 con 0x55a86d4f63d0 2024-08-09T14:51:47.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3bb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:51:47.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-08-09T14:51:47.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3bb7fe640 10 monclient: dump: 2024-08-09T14:51:47.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: epoch 1 2024-08-09T14:51:47.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:51:47.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:51:47.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:51:47.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: min_mon_release 19 (squid) 2024-08-09T14:51:47.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: election_strategy: 1 2024-08-09T14:51:47.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:51:47.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:51:47.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:51:47.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3cb6c6780 5 monclient: authenticate success, global_id 4364 2024-08-09T14:51:47.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3cb6c6780 10 monclient: get_monmap_and_config success 2024-08-09T14:51:47.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.327+0000 7fc3cb6c6780 4 set_mon_vals no callback set 2024-08-09T14:51:47.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.328+0000 7fc3cb6c6780 10 monclient: shutdown 2024-08-09T14:51:47.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.328+0000 7fc3cb6c6780 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:51:47.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.328+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/2531537878 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a86d4f63d0 msgr2=0x55a86d4f8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.328+0000 7fc3cb6c6780 1 --2- 172.21.15.155:0/2531537878 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a86d4f63d0 0x55a86d4f8850 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc001e00 tx=0x7fc3bc00b670 comp rx=0 tx=0).stop 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.328+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/2531537878 shutdown_connections 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.328+0000 7fc3cb6c6780 1 --2- 172.21.15.155:0/2531537878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a86d461130 0x55a86d45fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.328+0000 7fc3cb6c6780 1 --2- 172.21.15.155:0/2531537878 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a86d4f63d0 0x55a86d4f8850 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.328+0000 7fc3cb6c6780 1 --2- 172.21.15.155:0/2531537878 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a86d4642e0 0x55a86d461ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.328+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/2531537878 >> 172.21.15.155:0/2531537878 conn(0x55a86d46a2b0 msgr2=0x55a86d4693b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.328+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/2531537878 shutdown_connections 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.328+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/2531537878 wait complete. 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.328+0000 7fc3cb6c6780 10 monclient: build_initial_monmap 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.329+0000 7fc3cb6c6780 10 monclient: monmap: 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: epoch 0 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: last_changed 0.000000 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: created 0.000000 2024-08-09T14:51:47.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: min_mon_release 0 (unknown) 2024-08-09T14:51:47.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: election_strategy: 1 2024-08-09T14:51:47.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.noname-a 2024-08-09T14:51:47.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.noname-b 2024-08-09T14:51:47.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-c 2024-08-09T14:51:47.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.329+0000 7fc3cb6c6780 1 librados: starting msgr at 2024-08-09T14:51:47.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.329+0000 7fc3cb6c6780 1 librados: starting objecter 2024-08-09T14:51:47.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.329+0000 7fc3cb6c6780 1 Processor -- start 2024-08-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_stat.log 2024-08-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_stat: /' 2024-08-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.475 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.476 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.477 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.477 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.477 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.329+0000 7fc3cb6c6780 1 -- start start 2024-08-09T14:51:47.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.329+0000 7fc3cb6c6780 1 librados: setting wanted keys 2024-08-09T14:51:47.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.329+0000 7fc3cb6c6780 1 librados: calling monclient init 2024-08-09T14:51:47.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.329+0000 7fc3cb6c6780 10 monclient: init 2024-08-09T14:51:47.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 10 monclient: _reopen_session rank -1 2024-08-09T14:51:47.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 10 monclient: _add_conns ranks=[2,1,0] 2024-08-09T14:51:47.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a86d461130 0x55a86d4f4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 10 monclient(hunting): picked mon.noname-c con 0x55a86d461130 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a86d4642e0 0x55a86d4f6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 10 monclient(hunting): picked mon.noname-b con 0x55a86d4642e0 addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-08-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 1 --2- >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a86d595580 0x55a86d4f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 10 monclient(hunting): picked mon.noname-a con 0x55a86d595580 addr [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] 2024-08-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 10 monclient(hunting): start opening mon connection 2024-08-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 1 -- --> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] -- mon_getmap magic: 0 -- 0x55a86d4f8d90 con 0x55a86d595580 2024-08-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 10 monclient(hunting): start opening mon connection 2024-08-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a86d4edac0 con 0x55a86d4642e0 2024-08-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 10 monclient(hunting): start opening mon connection 2024-08-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x55a86d461810 con 0x55a86d461130 2024-08-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 10 monclient(hunting): _renew_subs 2024-08-09T14:51:47.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.330+0000 7fc3cb6c6780 10 monclient(hunting): authenticate will time out at 1314.367562s 2024-08-09T14:51:47.479 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.335+0000 7fc3c89c1640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a86d4642e0 0x55a86d4f6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:51:47.479 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.336+0000 7fc3c89c1640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a86d4642e0 0x55a86d4f6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.155:56400/0 (socket says 172.21.15.155:56400) 2024-08-09T14:51:47.479 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.336+0000 7fc3c89c1640 1 -- 172.21.15.155:0/3100773278 learned_addr learned my addr 172.21.15.155:0/3100773278 (peer_addr_for_me v2:172.21.15.155:0/0) 2024-08-09T14:51:47.479 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.336+0000 7fc3c89c1640 10 monclient(hunting): get_auth_request con 0x55a86d4642e0 auth_method 0 2024-08-09T14:51:47.485 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_cmd_pp 2024-08-09T14:51:47.485 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_cmd_pp' 2024-08-09T14:51:47.485 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.485 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.485 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.485 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.485 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.486 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.486 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.486 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.486 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.486 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.486 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.486 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.486 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.486 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.486 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.486 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.486 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.486 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_cmd_pp 2024-08-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.487 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.488 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.490 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.490 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.490 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.490 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.490 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.490 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.490 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.490 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.490 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_cmd_pp 2024-08-09T14:51:47.490 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41891 2024-08-09T14:51:47.490 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_cmd_pp on pid 41891' 2024-08-09T14:51:47.490 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.336+000test api_cmd_pp on pid 41891 2024-08-09T14:51:47.497 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41891 2024-08-09T14:51:47.497 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.497 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.497 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.498 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.498 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.498 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.498 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.498 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.498 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.498 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.498 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.498 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.498 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.498 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.498 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.499 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.499 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.499 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.499 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.499 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.499 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.499 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.499 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.499 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.499 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.499 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.499 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.499 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.500 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.500 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-08-09T14:51:47.500 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41828/exe 2024-08-09T14:51:47.500 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.500 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.500 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.500 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.500 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.500 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.500 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.501 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.501 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.501 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.501 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.501 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.501 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.501 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.501 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.501 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.501 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.501 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.501 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_service 2024-08-09T14:51:47.501 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.502 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.502 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.502 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.502 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.502 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_service' 2024-08-09T14:51:47.502 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.502 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.502 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.502 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.503 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.503 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.503 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.503 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.503 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.503 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.503 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.503 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.503 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.503 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.503 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.503 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.503 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.504 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.504 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.504 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.504 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.504 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.504 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.504 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.504 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.504 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.504 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.504 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.505 INFO:tasks.workunit.client.0.smithi155.stdout:0 7fc3c89c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-08-09T14:51:47.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.336+0000 7fc3c89c1640 10 monclient(hunting): _init_auth method 2 2024-08-09T14:51:47.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.336+0000 7fc3c89c1640 10 monclient(hunting): _init_auth creating new auth 2024-08-09T14:51:47.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.337+0000 7fc3c89c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-08-09T14:51:47.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.337+0000 7fc3c89c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-08-09T14:51:47.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.337+0000 7fc3c89c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-08-09T14:51:47.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.340+0000 7fc3c89c1640 10 monclient(hunting): handle_auth_done global_id 4435 payload 824 2024-08-09T14:51:47.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.340+0000 7fc3c89c1640 1 -- 172.21.15.155:0/3100773278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a86d461130 msgr2=0x55a86d4f4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:51:47.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.340+0000 7fc3c89c1640 1 --2- 172.21.15.155:0/3100773278 >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x55a86d461130 0x55a86d4f4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:51:47.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.340+0000 7fc3c89c1640 1 -- 172.21.15.155:0/3100773278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a86d595580 msgr2=0x55a86d4f2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:51:47.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.340+0000 7fc3c89c1640 1 --2- 172.21.15.155:0/3100773278 >> [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] conn(0x55a86d595580 0x55a86d4f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:51:47.506 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.340+0000 7fc3c89c1640 10 monclient: _finish_hunting 0 2024-08-09T14:51:47.506 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.340+0000 7fc3c89c1640 1 monclient: found mon.noname-b 2024-08-09T14:51:47.506 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.340+0000 7fc3c89c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:51:47.506 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.340+0000 7fc3c89c1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.194:3300/0 2024-08-09T14:51:47.506 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.340+0000 7fc3c89c1640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a86d4f3fc0 con 0x55a86d4642e0 2024-08-09T14:51:47.506 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.350+0000 7fc3c89c1640 1 --2- 172.21.15.155:0/3100773278 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a86d4642e0 0x55a86d4f6b00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00b6b0 tx=0x7fc3bc000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:51:47.506 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.350+0000 7fc3ba7fc640 10 client.?.objecter ms_handle_connect 0x55a86d4642e0 2024-08-09T14:51:47.506 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.350+0000 7fc3ba7fc640 10 client.?.objecter resend_mon_ops 2024-08-09T14:51:47.506 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.365+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc020070 con 0x55a86d4642e0 2024-08-09T14:51:47.506 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.365+0000 7fc3ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:51:47.506 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.365+0000 7fc3ba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-08-09T14:51:47.506 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.365+0000 7fc3ba7fc640 10 monclient: dump: 2024-08-09T14:51:47.506 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: epoch 1 2024-08-09T14:51:47.507 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:51:47.507 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:51:47.507 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:51:47.507 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: min_mon_release 19 (squid) 2024-08-09T14:51:47.509 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.509 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.509 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.510 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.510 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.510 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.510 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.510 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.510 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.510 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.510 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_service 2024-08-09T14:51:47.510 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-08-09T14:51:47.510 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-08-09T14:51:47.510 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.510 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.510 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.511 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.511 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.511 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.511 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.511 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.511 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.511 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.511 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.511 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_service 2024-08-09T14:51:47.511 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.511 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.511 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.511 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.511 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41949 2024-08-09T14:51:47.512 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_service on pid 41949' 2024-08-09T14:51:47.512 INFO:tasks.workunit.client.0.smithi155.stdout: test api_service on pid 41949 2024-08-09T14:51:47.520 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41949 2024-08-09T14:51:47.520 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.520 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.520 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.520 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.520 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.520 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.520 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-08-09T14:51:47.521 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.521 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.521 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.521 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.521 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.521 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.521 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.521 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.521 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_service_pp 2024-08-09T14:51:47.521 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_service_pp' 2024-08-09T14:51:47.521 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.521 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.521 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.522 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.522 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.522 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.522 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41844/exe 2024-08-09T14:51:47.522 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.522 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.522 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.522 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.522 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.522 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.522 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.522 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.522 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.524 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.524 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.524 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.524 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.524 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.524 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.524 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.524 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.524 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.524 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.524 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.524 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.524 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.524 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.525 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.525 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.525 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.525 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.525 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.525 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.525 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.525 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.525 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.525 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.525 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.525 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.525 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.525 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.526 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.526 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.526 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.526 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.526 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-08-09T14:51:47.526 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.526 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.526 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.526 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.526 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.526 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.526 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41811/exe 2024-08-09T14:51:47.526 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_service_pp 2024-08-09T14:51:47.527 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.527 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.527 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.527 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.527 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.527 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.527 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.527 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.527 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.527 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.527 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.527 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.527 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.527 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.528 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.528 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.528 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.528 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.528 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_cmd: /' 2024-08-09T14:51:47.528 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.528 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.528 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.528 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-08-09T14:51:47.528 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.528 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_cmd.log 2024-08-09T14:51:47.528 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_service_pp 2024-08-09T14:51:47.528 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41977 2024-08-09T14:51:47.529 INFO:tasks.workunit.client.0.smithi155.stdout:test api_service_pp on pid 41977 2024-08-09T14:51:47.534 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_service_pp on pid 41977' 2024-08-09T14:51:47.534 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41977 2024-08-09T14:51:47.534 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.534 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.534 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-08-09T14:51:47.534 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-08-09T14:51:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-08-09T14:51:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_watch_notify: /' 2024-08-09T14:51:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_c_write_operations 2024-08-09T14:51:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_c_write_operations' 2024-08-09T14:51:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.537 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.537 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.537 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.537 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.537 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.537 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.537 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.537 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.537 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.537 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.537 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.537 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.537 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_c_write_operations 2024-08-09T14:51:47.537 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.538 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.539 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.539 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.539 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.539 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_c_write_operations 2024-08-09T14:51:47.539 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.539 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.539 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41989 2024-08-09T14:51:47.539 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_c_write_operations on pid 41989' 2024-08-09T14:51:47.539 INFO:tasks.workunit.client.0.smithi155.stdout:test api_c_write_operations on pid 41989 2024-08-09T14:51:47.546 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=41989 2024-08-09T14:51:47.546 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.546 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-08-09T14:51:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_c_read_operations 2024-08-09T14:51:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_c_read_operations' 2024-08-09T14:51:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.550 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.550 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_c_read_operations 2024-08-09T14:51:47.550 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.550 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.550 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.550 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.550 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.550 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.550 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.550 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.550 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.550 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.550 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.552 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.552 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.552 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.552 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.552 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.552 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.552 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.552 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.552 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.552 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.552 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.552 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.552 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.553 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.553 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.553 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.553 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.553 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.553 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.553 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.553 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.553 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.553 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.553 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.553 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.553 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.553 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.554 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.554 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.554 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.554 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.554 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_c_read_operations 2024-08-09T14:51:47.554 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42014 2024-08-09T14:51:47.554 INFO:tasks.workunit.client.0.smithi155.stdout:test api_c_read_operations on pid 42014 2024-08-09T14:51:47.557 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_c_read_operations on pid 42014' 2024-08-09T14:51:47.557 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42014 2024-08-09T14:51:47.557 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.557 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.557 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.557 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-08-09T14:51:47.557 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.557 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.557 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.557 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.557 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.558 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.558 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.558 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.558 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.558 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.558 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.558 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.558 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.558 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.558 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.558 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.558 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.558 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.559 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.559 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.559 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.559 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.559 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.559 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.559 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.559 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: election_strategy: 1 2024-08-09T14:51:47.559 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:51:47.559 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:51:47.559 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:51:47.560 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.560 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.365+0000 7fc3ba7fc640 10 monclient: _finish_auth 0 2024-08-09T14:51:47.560 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.365+0000 7fc3ba7fc640 10 monclient: _check_auth_tickets 2024-08-09T14:51:47.560 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.365+0000 7fc3ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:51:47.560 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.365+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3bc00be40 con 0x55a86d4642e0 2024-08-09T14:51:47.560 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.365+0000 7fc3ba7fc640 10 monclient: handle_config config(0 keys) 2024-08-09T14:51:47.560 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.365+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc01c6a0 con 0x55a86d4642e0 2024-08-09T14:51:47.560 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.365+0000 7fc3ba7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-08-09T14:51:47.560 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.365+0000 7fc3ba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-08-09T14:51:47.560 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.365+0000 7fc3ba7fc640 10 monclient: dump: 2024-08-09T14:51:47.560 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: epoch 1 2024-08-09T14:51:47.560 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:51:47.560 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: last_changed 2024-08-09T14:48:51.873792+0000 2024-08-09T14:51:47.561 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: created 2024-08-09T14:48:51.873792+0000 2024-08-09T14:51:47.561 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: min_mon_release 19 (squid) 2024-08-09T14:51:47.561 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: election_strategy: 1 2024-08-09T14:51:47.561 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 0: [v2:172.21.15.155:3300/0,v1:172.21.15.155:6789/0] mon.a 2024-08-09T14:51:47.561 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-08-09T14:51:47.561 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.c 2024-08-09T14:51:47.561 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.561 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.366+0000 7fc3cb6c6780 5 monclient: authenticate success, global_id 4435 2024-08-09T14:51:47.561 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.366+0000 7fc3cb6c6780 10 monclient: _renew_subs 2024-08-09T14:51:47.561 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.366+0000 7fc3cb6c6780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:51:47.561 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.366+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55a86d4f42d0 con 0x55a86d4642e0 2024-08-09T14:51:47.561 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.366+0000 7fc3cb6c6780 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:51:47.561 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.366+0000 7fc3cb6c6780 10 monclient: _renew_subs 2024-08-09T14:51:47.562 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.366+0000 7fc3cb6c6780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:51:47.562 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.366+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a86d4f7040 con 0x55a86d4642e0 2024-08-09T14:51:47.562 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.366+0000 7fc3cb6c6780 1 librados: init done 2024-08-09T14:51:47.562 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.366+0000 7fc3cb6c6780 10 librados: wait_for_osdmap waiting 2024-08-09T14:51:47.562 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.366+0000 7fc3bb7fe640 4 set_mon_vals no callback set 2024-08-09T14:51:47.562 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.399+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fc3bc023070 con 0x55a86d4642e0 2024-08-09T14:51:47.573 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s api_cls_remote_reads 2024-08-09T14:51:47.573 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' api_cls_remote_reads' 2024-08-09T14:51:47.573 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.573 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.573 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.573 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.573 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41891/exe 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.575 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.576 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.577 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.578 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.578 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.578 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.578 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.578 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.578 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.578 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.578 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.578 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.578 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.578 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.578 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.578 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.578 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.579 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.579 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.579 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.579 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.579 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.579 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.579 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41949/exe 2024-08-09T14:51:47.579 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.579 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.579 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.579 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.579 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.579 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.580 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo api_cls_remote_reads 2024-08-09T14:51:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-08-09T14:51:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.581 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.584 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.584 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=api_cls_remote_reads 2024-08-09T14:51:47.584 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.584 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.584 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.584 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.584 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42047 2024-08-09T14:51:47.584 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.584 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.400+0000 7fc3ba7fc640 1 --2- 172.21.15.155:0/3100773278 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fc3ac06ba30 0x7fc3ac06def0 unknown :-1 s=NONE pgstest api_cls_remote_reads on pid 42047 2024-08-09T14:51:47.595 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test api_cls_remote_reads on pid 42047' 2024-08-09T14:51:47.595 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.595 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42047 2024-08-09T14:51:47.595 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.595 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.595 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.596 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-08-09T14:51:47.596 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.596 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.596 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.596 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.596 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.596 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s list_parallel 2024-08-09T14:51:47.596 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.596 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' list_parallel' 2024-08-09T14:51:47.596 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.596 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.596 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.596 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.596 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.597 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.597 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.597 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.597 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.597 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.597 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.597 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.597 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.597 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.597 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.597 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.597 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.597 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.598 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.599 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.600 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.600 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.600 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.600 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.600 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.600 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.600 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.600 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.600 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.600 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.600 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.600 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.600 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.601 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.601 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.601 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.601 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.601 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.601 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.601 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.601 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.601 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.601 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_service: /' 2024-08-09T14:51:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_service.log 2024-08-09T14:51:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-08-09T14:51:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.603 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.603 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.603 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.603 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.603 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.603 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.603 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.603 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.603 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.603 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.603 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.603 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.604 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.604 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.604 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.604 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.604 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.604 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.604 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.604 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.604 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.604 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-08-09T14:51:47.604 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.604 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.604 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.606 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo list_parallel 2024-08-09T14:51:47.606 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.606 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.606 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41977/exe 2024-08-09T14:51:47.606 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.606 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.606 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.606 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.606 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.606 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.606 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.606 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.606 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.607 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=list_parallel 2024-08-09T14:51:47.607 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.607 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42091 2024-08-09T14:51:47.607 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test list_parallel on pid 42091' 2024-08-09T14:51:47.607 INFO:tasks.workunit.client.0.smithi155.stdout:test list_parallel on pid 42091 2024-08-09T14:51:47.611 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42091 2024-08-09T14:51:47.611 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.611 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.611 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.611 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.611 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s open_pools_parallel 2024-08-09T14:51:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' open_pools_parallel' 2024-08-09T14:51:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo open_pools_parallel 2024-08-09T14:51:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-08-09T14:51:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-08-09T14:51:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_service_pp: /' 2024-08-09T14:51:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42014/exe 2024-08-09T14:51:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-08-09T14:51:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=open_pools_parallel 2024-08-09T14:51:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42116 2024-08-09T14:51:47.615 INFO:tasks.workunit.client.0.smithi155.stdout:test open_pools_parallel on pid 42116 2024-08-09T14:51:47.625 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test open_pools_parallel on pid 42116' 2024-08-09T14:51:47.625 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42116 2024-08-09T14:51:47.625 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-08-09T14:51:47.625 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.625 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.625 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-08-09T14:51:47.625 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.625 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.625 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.625 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.625 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.625 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.626 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.626 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.626 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.626 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.626 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.626 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.626 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.626 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.626 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.626 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.626 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.626 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.626 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.626 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.627 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.627 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.627 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s delete_pools_parallel 2024-08-09T14:51:47.627 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.627 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.627 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.627 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.627 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.627 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.627 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.627 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.627 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.627 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.628 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' delete_pools_parallel' 2024-08-09T14:51:47.628 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.628 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.628 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.628 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.628 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.628 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.628 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.628 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.628 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.628 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.628 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.628 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.629 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.629 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.629 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.629 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.629 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.629 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.629 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.629 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.629 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.629 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.629 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.629 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.629 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.630 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.630 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.630 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.630 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.630 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.630 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.630 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.630 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.630 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.630 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.630 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.630 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.630 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.631 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.631 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.631 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.631 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.631 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.631 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.631 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.631 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.631 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.631 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.631 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.631 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.631 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.632 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.632 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.632 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.632 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.632 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.632 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.632 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.632 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.632 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.632 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.632 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.632 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.632 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo delete_pools_parallel 2024-08-09T14:51:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-08-09T14:51:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/41989/exe 2024-08-09T14:51:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-08-09T14:51:47.634 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-08-09T14:51:47.634 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.634 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.634 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.634 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.634 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.634 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.634 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=delete_pools_parallel 2024-08-09T14:51:47.634 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42135 2024-08-09T14:51:47.634 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test delete_pools_parallel on pid 42135' 2024-08-09T14:51:47.634 INFO:tasks.workunit.client.0.smithi155.stdout:test delete_pools_parallel on pid 42135 2024-08-09T14:51:47.643 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42135 2024-08-09T14:51:47.643 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-09T14:51:47.643 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.643 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.643 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.643 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.643 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.643 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.643 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.643 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.643 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.643 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.644 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.644 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-08-09T14:51:47.644 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.644 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.644 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.644 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.644 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.644 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.644 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.644 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.644 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.644 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.644 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.645 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.645 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.645 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.645 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.645 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.645 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.645 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.645 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.645 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.645 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-08-09T14:51:47.645 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s cls 2024-08-09T14:51:47.645 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' cls' 2024-08-09T14:51:47.645 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo cls 2024-08-09T14:51:47.646 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.646 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-08-09T14:51:47.646 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.646 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.646 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.646 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.646 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.646 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.646 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.646 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.646 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.646 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.646 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.647 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.647 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.647 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.647 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.647 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.647 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.647 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.647 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.647 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.647 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.647 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.647 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.647 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.649 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.649 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.649 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.649 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42047/exe 2024-08-09T14:51:47.649 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.649 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.649 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.649 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.649 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.649 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.649 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.649 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.649 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.649 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-08-09T14:51:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=cls 2024-08-09T14:51:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.651 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42161 2024-08-09T14:51:47.651 INFO:tasks.workunit.client.0.smithi155.stdout:test cls on pid 42161 2024-08-09T14:51:47.659 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test cls on pid 42161' 2024-08-09T14:51:47.659 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42161 2024-08-09T14:51:47.659 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-09T14:51:47.659 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.659 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.659 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.659 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.659 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.659 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.659 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.659 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.659 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s cmd 2024-08-09T14:51:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' cmd' 2024-08-09T14:51:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-08-09T14:51:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-08-09T14:51:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-08-09T14:51:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-08-09T14:51:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.665 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.665 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.665 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.665 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.665 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.665 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.665 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.665 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.665 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.665 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.665 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.665 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.665 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.666 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.666 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.666 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo cmd 2024-08-09T14:51:47.666 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.666 INFO:tasks.workunit.client.0.smithi155.stdout:test cmd on pid 42207 2024-08-09T14:51:47.681 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.681 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.681 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=cmd 2024-08-09T14:51:47.681 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.681 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.682 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.682 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.682 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42207 2024-08-09T14:51:47.682 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test cmd on pid 42207' 2024-08-09T14:51:47.682 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.682 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42207 2024-08-09T14:51:47.682 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-09T14:51:47.682 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.682 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.682 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.682 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.682 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.683 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.683 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.683 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.683 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.683 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.683 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.683 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.683 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-08-09T14:51:47.683 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.683 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s handler_error 2024-08-09T14:51:47.683 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' handler_error' 2024-08-09T14:51:47.683 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.683 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.687 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.687 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.687 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.687 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.687 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.687 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.687 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.687 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.687 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.687 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.687 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.687 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.687 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.688 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.688 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.688 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.688 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.688 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.688 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.688 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.688 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.688 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.688 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.688 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.688 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.688 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.689 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.689 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.689 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.689 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.689 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.689 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.689 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.689 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.689 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.689 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.689 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.689 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.689 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.689 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.690 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.690 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.690 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.690 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.690 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.690 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.690 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.690 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.690 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.690 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.690 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.690 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.690 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.690 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.691 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.691 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.691 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.691 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.691 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.691 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.691 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.691 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.691 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.691 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.691 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.691 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.691 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.692 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.692 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.692 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.692 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.692 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.692 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.692 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.692 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.692 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.692 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo handler_error 2024-08-09T14:51:47.692 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.692 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42091/exe 2024-08-09T14:51:47.692 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.693 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.693 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.693 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.693 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.693 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.693 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.693 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.693 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.693 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.693 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.693 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.693 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.693 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.693 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.694 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.694 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.694 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.694 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.694 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.694 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.694 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.694 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.694 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.694 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.694 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.694 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.695 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.695 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.695 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.695 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.695 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.695 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.695 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.695 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.695 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.695 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.695 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.695 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.695 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.696 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.696 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.696 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.696 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.696 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.696 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.696 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.696 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.696 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.696 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.696 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.696 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.696 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.696 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.697 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.697 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.697 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.697 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.697 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.697 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.697 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.697 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.697 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.697 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.697 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.697 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.697 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.697 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.698 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_list_parallel.log 2024-08-09T14:51:47.698 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=handler_error 2024-08-09T14:51:47.698 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42277 2024-08-09T14:51:47.698 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test handler_error on pid 42277' 2024-08-09T14:51:47.698 INFO:tasks.workunit.client.0.smithi155.stdout:test handler_error on pid 42277 2024-08-09T14:51:47.702 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42277 2024-08-09T14:51:47.702 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-09T14:51:47.702 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.702 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-08-09T14:51:47.702 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s io 2024-08-09T14:51:47.702 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' io' 2024-08-09T14:51:47.702 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.702 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.702 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.702 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.702 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.703 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.703 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.703 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.703 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.703 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo io 2024-08-09T14:51:47.703 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.703 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.703 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.703 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-08-09T14:51:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42116/exe 2024-08-09T14:51:47.705 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=io 2024-08-09T14:51:47.705 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42299 2024-08-09T14:51:47.705 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test io on pid 42299' 2024-08-09T14:51:47.705 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.705 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.705 INFO:tasks.workunit.client.0.smithi155.stdout:test io on pid 42299 2024-08-09T14:51:47.718 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.718 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42299 2024-08-09T14:51:47.719 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-09T14:51:47.719 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.719 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.719 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.719 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.719 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s ec_io 2024-08-09T14:51:47.719 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.719 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.719 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-08-09T14:51:47.719 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' ec_io' 2024-08-09T14:51:47.719 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.719 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.719 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.719 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.720 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.720 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ list_parallel: /' 2024-08-09T14:51:47.720 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo ec_io 2024-08-09T14:51:47.720 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.720 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.720 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.720 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.720 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.720 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.720 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.720 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.720 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.720 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.721 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.721 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.721 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.721 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.721 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.721 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.721 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.721 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.721 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.721 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.721 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-08-09T14:51:47.721 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.721 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.722 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.722 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.722 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.722 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.722 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.722 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.722 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.722 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.722 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.722 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.722 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.722 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.722 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42207/exe 2024-08-09T14:51:47.725 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.726 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42135/exe 2024-08-09T14:51:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.727 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.728 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.728 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.728 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.728 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.728 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.728 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.728 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.728 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.728 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.728 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.728 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.728 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.728 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.729 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.729 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.729 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.729 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.729 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.729 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.729 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.729 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.729 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-08-09T14:51:47.729 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.729 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.729 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.729 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.729 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.730 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=ec_io 2024-08-09T14:51:47.730 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.730 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.730 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.730 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42327 2024-08-09T14:51:47.730 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test ec_io on pid 42327' 2024-08-09T14:51:47.730 INFO:tasks.workunit.client.0.smithi155.stdout:test ec_io on pid 42327 2024-08-09T14:51:47.741 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42327 2024-08-09T14:51:47.741 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-09T14:51:47.741 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.741 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.741 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.741 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.741 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.741 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.742 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.742 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.742 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.742 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.742 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.742 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_neorados_cmd 2024-08-09T14:51:47.742 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.742 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.742 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.742 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.742 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-08-09T14:51:47.742 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.742 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-08-09T14:51:47.742 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.743 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_neorados_cmd.log 2024-08-09T14:51:47.743 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ cmd: /' 2024-08-09T14:51:47.743 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.743 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.743 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.743 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.743 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.743 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.743 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.743 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.743 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.743 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.743 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.744 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.744 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.744 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.744 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.744 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.744 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.744 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.744 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.744 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.744 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.744 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.744 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.744 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.745 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.746 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.746 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.746 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-08-09T14:51:47.746 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-08-09T14:51:47.746 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-08-09T14:51:47.746 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s list 2024-08-09T14:51:47.746 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' list' 2024-08-09T14:51:47.746 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.746 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.746 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.746 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.746 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42161/exe 2024-08-09T14:51:47.746 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.746 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.747 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.748 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.749 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.750 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.750 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.750 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.750 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.750 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ cls: /' 2024-08-09T14:51:47.750 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.750 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.750 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.750 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_neorados_cls 2024-08-09T14:51:47.750 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_neorados_cls.log 2024-08-09T14:51:47.750 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo list 2024-08-09T14:51:47.750 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.750 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=list 2024-08-09T14:51:47.750 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42369 2024-08-09T14:51:47.751 INFO:tasks.workunit.client.0.smithi155.stdout:test list on pid 42369 2024-08-09T14:51:47.759 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test list on pid 42369' 2024-08-09T14:51:47.759 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42369 2024-08-09T14:51:47.759 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-09T14:51:47.759 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.759 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.759 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.759 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.759 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.760 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.760 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.760 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.760 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.760 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.760 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-08-09T14:51:47.760 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s ec_list 2024-08-09T14:51:47.760 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' ec_list' 2024-08-09T14:51:47.760 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.760 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.760 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.760 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.760 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.760 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.761 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo ec_list 2024-08-09T14:51:47.761 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.761 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.761 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.761 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.761 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.761 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.761 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.761 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.761 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.761 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.761 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.761 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.762 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.762 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.762 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.762 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.762 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.762 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.762 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.762 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.762 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.762 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.762 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.762 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.762 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.763 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.763 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.763 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.763 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.763 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.763 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.763 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.763 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.763 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.763 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.763 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.763 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.763 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.764 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.764 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.764 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.764 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.764 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.764 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.764 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.764 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.764 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.764 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.764 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.764 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.764 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.764 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.765 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.765 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.765 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.765 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.765 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.765 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.765 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.765 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.765 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.765 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.765 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.765 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.765 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.766 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.766 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.766 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.766 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.766 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.766 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.766 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.766 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.766 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.766 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.766 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.766 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.766 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.766 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.767 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.767 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.767 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.767 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.767 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=ec_list 2024-08-09T14:51:47.767 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42431 2024-08-09T14:51:47.767 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.767 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.767 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test ec_list on pid 42431' 2024-08-09T14:51:47.767 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.767 INFO:tasks.workunit.client.0.smithi155.stdout:test ec_list on pid 42431 2024-08-09T14:51:47.774 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42431 2024-08-09T14:51:47.774 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-09T14:51:47.774 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.774 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.774 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.775 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.775 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.775 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.775 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.775 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s misc 2024-08-09T14:51:47.775 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' misc' 2024-08-09T14:51:47.775 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo misc 2024-08-09T14:51:47.775 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.775 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-08-09T14:51:47.775 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.775 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.775 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.775 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.775 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.778 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.778 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.778 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.778 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.778 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.778 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.778 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.778 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.778 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.778 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.778 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.778 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.778 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.779 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.779 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.779 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.779 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.779 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.779 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.779 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.779 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.779 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.779 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.779 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.779 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.779 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.780 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.780 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42277/exe 2024-08-09T14:51:47.780 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.780 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=misc 2024-08-09T14:51:47.780 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.780 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.780 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.780 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.780 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.780 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.780 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42452 2024-08-09T14:51:47.781 INFO:tasks.workunit.client.0.smithi155.stdout:test misc on pid 42452 2024-08-09T14:51:47.788 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test misc on pid 42452' 2024-08-09T14:51:47.789 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42452 2024-08-09T14:51:47.789 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-09T14:51:47.789 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.789 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.789 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.789 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.789 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.789 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.789 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.789 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.789 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.789 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s pool 2024-08-09T14:51:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-08-09T14:51:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' pool' 2024-08-09T14:51:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.792 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.792 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.792 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.792 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.792 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.792 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo pool 2024-08-09T14:51:47.792 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.792 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.792 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.792 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.792 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.792 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.792 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.793 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.793 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.793 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.793 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.793 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.793 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.793 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.793 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.793 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.793 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.793 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.793 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.793 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.793 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.794 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.794 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.794 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.794 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.794 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_neorados_handler_error.log 2024-08-09T14:51:47.794 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.794 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.794 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.794 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.794 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.794 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.794 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.794 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.795 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.795 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.795 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ handler_error: /' 2024-08-09T14:51:47.795 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.795 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.795 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.795 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.795 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.795 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.795 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.795 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.795 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.795 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.795 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.796 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.796 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.796 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.796 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.796 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.796 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.796 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.796 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.796 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.796 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.796 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.796 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.796 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.797 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.797 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.797 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.797 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.797 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.797 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.797 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.797 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.797 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.797 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.797 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.797 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.797 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=pool 2024-08-09T14:51:47.797 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42474 2024-08-09T14:51:47.798 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test pool on pid 42474' 2024-08-09T14:51:47.798 INFO:tasks.workunit.client.0.smithi155.stdout:test pool on pid 42474 2024-08-09T14:51:47.807 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.807 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42474 2024-08-09T14:51:47.808 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-09T14:51:47.808 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.808 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-08-09T14:51:47.808 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_neorados_handler_error 2024-08-09T14:51:47.808 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s read_operations 2024-08-09T14:51:47.808 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.808 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' read_operations' 2024-08-09T14:51:47.808 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.808 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.808 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.808 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.808 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.808 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo read_operations 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.810 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.810 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.810 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.810 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.810 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.810 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.810 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.810 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.810 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.810 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.810 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.810 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.811 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.812 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.812 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.812 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.812 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.812 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.812 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.812 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.812 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.812 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.812 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.812 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.812 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.812 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=read_operations 2024-08-09T14:51:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42497 2024-08-09T14:51:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.815 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.816 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.816 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test read_operations on pid 42497' 2024-08-09T14:51:47.816 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.816 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.816 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.816 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.816 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.816 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.816 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.816 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.816 INFO:tasks.workunit.client.0.smithi155.stdout:test read_operations on pid 42497 2024-08-09T14:51:47.826 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42497 2024-08-09T14:51:47.827 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-08-09T14:51:47.827 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-09T14:51:47.827 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.827 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.827 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.827 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.827 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.827 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.827 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.827 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.827 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.827 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.827 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42327/exe 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s snapshots 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' snapshots' 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42369/exe 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.828 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.829 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.830 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.830 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.830 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.830 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.830 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.830 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.830 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.830 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.830 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.830 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.830 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.830 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.830 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.830 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42299/exe 2024-08-09T14:51:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.832 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.833 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.833 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.833 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.833 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.833 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.833 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.833 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.833 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.833 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.833 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.833 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.833 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.833 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.833 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.834 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.834 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.834 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.834 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.834 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.834 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.834 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.834 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.834 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.834 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.834 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.834 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.834 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.835 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.835 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.835 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.835 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.835 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.835 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.835 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.835 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.835 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.835 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.835 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.835 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.835 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo snapshots 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_neorados_list.log 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ list: /' 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_neorados_ec_io.log 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_neorados_list 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.836 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.837 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.837 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.837 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_neorados_ec_io 2024-08-09T14:51:47.837 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.837 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.837 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.837 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=snapshots 2024-08-09T14:51:47.837 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42528 2024-08-09T14:51:47.837 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test snapshots on pid 42528' 2024-08-09T14:51:47.837 INFO:tasks.workunit.client.0.smithi155.stdout:test snapshots on pid 42528 2024-08-09T14:51:47.849 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42528 2024-08-09T14:51:47.849 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-09T14:51:47.850 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.850 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_neorados_io.log 2024-08-09T14:51:47.850 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s watch_notify 2024-08-09T14:51:47.850 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42431/exe 2024-08-09T14:51:47.850 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ ec_io: /' 2024-08-09T14:51:47.850 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_neorados_io 2024-08-09T14:51:47.850 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' watch_notify' 2024-08-09T14:51:47.850 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ io: /' 2024-08-09T14:51:47.850 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.850 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.850 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.850 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.851 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.851 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.851 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-08-09T14:51:47.851 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.851 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.851 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.851 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.851 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.851 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.851 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.851 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.851 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.851 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.852 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.852 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.852 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.852 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.852 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.852 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.852 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.852 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.852 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.852 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.852 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.852 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.852 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.852 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.853 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ ec_list: /' 2024-08-09T14:51:47.853 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.853 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.853 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.853 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.853 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.853 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.853 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.853 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.853 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.853 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.853 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.853 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.854 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.854 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.854 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_neorados_ec_list.log 2024-08-09T14:51:47.854 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.854 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.854 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_neorados_ec_list 2024-08-09T14:51:47.854 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.854 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.854 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.854 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.854 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.854 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.854 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.855 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.855 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.855 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.855 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.855 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.855 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.855 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.855 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.855 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.855 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.855 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.855 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.856 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.856 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.856 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.856 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.856 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.856 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.856 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.856 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.856 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.856 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.856 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.856 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.856 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.857 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.857 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.857 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.857 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.857 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.857 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.857 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.857 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.857 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.857 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo watch_notify 2024-08-09T14:51:47.857 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.857 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.857 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.857 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.859 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.860 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.861 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.861 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.861 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.861 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.861 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.861 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.861 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=watch_notify 2024-08-09T14:51:47.861 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42567 2024-08-09T14:51:47.861 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test watch_notify on pid 42567' 2024-08-09T14:51:47.861 INFO:tasks.workunit.client.0.smithi155.stdout:test watch_notify on pid 42567 2024-08-09T14:51:47.868 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42567 2024-08-09T14:51:47.868 INFO:tasks.workunit.client.0.smithi155.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-08-09T14:51:47.868 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.868 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.868 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.868 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-08-09T14:51:47.868 INFO:tasks.workunit.client.0.smithi155.stderr:++ printf %25s write_operations 2024-08-09T14:51:47.868 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.869 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.869 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.869 INFO:tasks.workunit.client.0.smithi155.stderr:+ r=' write_operations' 2024-08-09T14:51:47.869 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.869 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.869 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.869 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.869 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.869 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.869 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.869 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.869 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.869 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.870 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.870 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.870 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.870 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.870 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.870 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.870 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.870 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42452/exe 2024-08-09T14:51:47.870 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.870 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.870 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.870 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.870 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.871 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.871 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.871 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.871 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.871 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.871 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.871 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.871 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.871 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.871 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.871 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.871 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.871 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.871 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.872 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.872 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.872 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.872 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.872 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.872 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.872 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.872 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.872 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.872 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.872 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.872 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.872 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.873 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.873 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.873 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.873 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.873 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_neorados_misc.log 2024-08-09T14:51:47.873 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_neorados_misc 2024-08-09T14:51:47.873 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ misc: /' 2024-08-09T14:51:47.873 INFO:tasks.workunit.client.0.smithi155.stderr:++ echo write_operations 2024-08-09T14:51:47.873 INFO:tasks.workunit.client.0.smithi155.stderr:++ awk '{print $1}' 2024-08-09T14:51:47.873 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.873 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.873 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.873 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.873 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.874 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.874 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.874 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.874 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.874 INFO:tasks.workunit.client.0.smithi155.stderr:+ ff=write_operations 2024-08-09T14:51:47.874 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=42587 2024-08-09T14:51:47.874 INFO:tasks.workunit.client.0.smithi155.stderr:+ echo 'test write_operations on pid 42587' 2024-08-09T14:51:47.874 INFO:tasks.workunit.client.0.smithi155.stdout:test write_operations on pid 42587 2024-08-09T14:51:47.904 INFO:tasks.workunit.client.0.smithi155.stderr:+ pids[$f]=42587 2024-08-09T14:51:47.904 INFO:tasks.workunit.client.0.smithi155.stderr:+ ret=0 2024-08-09T14:51:47.904 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' 1 -eq 1 ']' 2024-08-09T14:51:47.904 INFO:tasks.workunit.client.0.smithi155.stderr:+ for t in "${!pids[@]}" 2024-08-09T14:51:47.904 INFO:tasks.workunit.client.0.smithi155.stderr:+ pid=41543 2024-08-09T14:51:47.904 INFO:tasks.workunit.client.0.smithi155.stderr:+ wait 41543 2024-08-09T14:51:47.904 INFO:tasks.workunit.client.0.smithi155.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-08-09T14:51:47.904 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.905 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.905 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.905 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.905 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.905 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.905 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.905 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.905 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.905 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.905 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.905 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.905 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.905 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.906 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.906 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.906 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.906 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -f /etc/bashrc ']' 2024-08-09T14:51:47.906 INFO:tasks.workunit.client.0.smithi155.stderr:+ . /etc/bashrc 2024-08-09T14:51:47.906 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.906 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -z '' ']' 2024-08-09T14:51:47.906 INFO:tasks.workunit.client.0.smithi155.stderr:++ BASHRCSOURCED=Y 2024-08-09T14:51:47.906 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.906 INFO:tasks.workunit.client.0.smithi155.stderr:++ shopt -q login_shell 2024-08-09T14:51:47.906 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.906 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.906 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.907 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.907 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.907 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.907 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.907 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.907 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.907 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.907 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.907 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.907 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.907 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.907 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.907 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.907 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.908 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.908 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.908 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.908 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.908 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.908 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.908 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.908 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.908 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.908 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.908 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.908 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.908 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.908 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.909 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.909 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.909 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.909 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.909 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.909 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.909 INFO:tasks.workunit.client.0.smithi155.stderr:+++ umask 2024-08-09T14:51:47.909 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.909 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.909 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.909 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.909 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.909 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.909 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' 0022 -eq 0 ']' 2024-08-09T14:51:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:++ SHELL=/bin/bash 2024-08-09T14:51:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-08-09T14:51:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/bash_completion.sh 2024-08-09T14:51:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-08-09T14:51:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.911 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.911 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-08-09T14:51:47.911 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.911 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/cobbler.sh 2024-08-09T14:51:47.911 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.911 INFO:tasks.workunit.client.0.smithi155.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-08-09T14:51:47.911 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.911 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-08-09T14:51:47.911 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.911 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorgrep.sh 2024-08-09T14:51:47.911 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.911 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.911 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.912 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.912 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.912 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.912 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.912 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.912 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.912 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.912 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.912 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.912 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.912 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.912 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.912 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.913 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.914 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.914 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.914 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.914 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.914 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.914 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.914 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.914 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.914 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.914 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.914 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.914 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.914 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.915 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.915 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.915 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.915 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.915 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.915 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.915 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.915 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.915 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.915 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.915 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.915 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.915 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.916 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.916 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.916 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42474/exe 2024-08-09T14:51:47.916 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.916 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.916 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.916 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.916 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.916 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.916 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.916 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.916 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.916 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.916 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.917 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.917 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.917 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.917 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.917 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.917 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.917 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.917 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42497/exe 2024-08-09T14:51:47.917 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.917 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.917 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.917 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.917 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.918 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.918 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.918 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.918 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.918 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.918 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.918 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.918 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.918 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.918 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.918 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.918 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.919 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.919 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.919 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.919 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.919 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.919 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.919 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.919 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.919 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.919 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.919 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.919 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.919 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.919 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.920 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.920 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.920 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.920 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.920 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.920 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.920 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.920 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'grep=grep --color=auto' 2024-08-09T14:51:47.920 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.920 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'egrep=egrep --color=auto' 2024-08-09T14:51:47.920 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.920 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.920 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-08-09T14:51:47.920 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.921 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.921 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.921 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-08-09T14:51:47.921 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.921 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorls.sh 2024-08-09T14:51:47.921 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' '!' -t 0 ']' 2024-08-09T14:51:47.921 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.921 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.921 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.921 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.921 INFO:tasks.workunit.client.0.smithi155.stderr:+++ return 2024-08-09T14:51:47.921 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.922 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-08-09T14:51:47.922 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.922 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-08-09T14:51:47.922 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export S_COLORS=auto 2024-08-09T14:51:47.922 INFO:tasks.workunit.client.0.smithi155.stderr:+++ S_COLORS=auto 2024-08-09T14:51:47.922 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.922 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_neorados_read_operations 2024-08-09T14:51:47.922 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-08-09T14:51:47.922 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.922 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-08-09T14:51:47.922 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.922 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ pool: /' 2024-08-09T14:51:47.922 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_neorados_pool.log 2024-08-09T14:51:47.922 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_neorados_pool 2024-08-09T14:51:47.923 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.923 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.923 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.923 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.923 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.923 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.923 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.923 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.923 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.923 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.923 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.923 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.923 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.924 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.925 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ read_operations: /' 2024-08-09T14:51:47.925 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.925 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.925 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.926 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.926 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.926 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.926 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.926 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.926 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.926 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.926 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.926 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.926 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.926 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_neorados_read_operations.log 2024-08-09T14:51:47.926 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42528/exe 2024-08-09T14:51:47.926 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.927 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.927 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.927 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.927 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.927 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.927 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.927 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.927 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.927 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.927 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.927 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-08-09T14:51:47.927 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-08-09T14:51:47.927 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.928 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.928 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.928 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.928 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.928 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.928 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-08-09T14:51:47.928 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.928 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-08-09T14:51:47.928 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.928 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-08-09T14:51:47.928 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-08-09T14:51:47.928 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.928 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.929 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.929 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.929 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.929 INFO:tasks.workunit.client.0.smithi155.stderr:+++ /usr/libexec/grepconf.sh -c 2024-08-09T14:51:47.929 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.929 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.929 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.929 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.929 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42567/exe 2024-08-09T14:51:47.929 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.929 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.929 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.929 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.929 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.930 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.930 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_neorados_snapshots 2024-08-09T14:51:47.930 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.930 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ snapshots: /' 2024-08-09T14:51:47.930 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.930 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.930 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.930 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.930 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.930 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.930 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.930 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.930 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.931 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.931 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_neorados_snapshots.log 2024-08-09T14:51:47.931 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_neorados_watch_notify 2024-08-09T14:51:47.931 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ watch_notify: /' 2024-08-09T14:51:47.931 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-08-09T14:51:47.931 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-08-09T14:51:47.931 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-08-09T14:51:47.931 INFO:tasks.workunit.client.0.smithi155.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-08-09T14:51:47.931 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.931 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-08-09T14:51:47.931 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.931 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/debuginfod.sh 2024-08-09T14:51:47.931 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-08-09T14:51:47.932 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.932 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-08-09T14:51:47.932 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.932 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/gawk.sh 2024-08-09T14:51:47.932 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.932 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-08-09T14:51:47.932 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.932 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/lang.sh 2024-08-09T14:51:47.932 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup 2024-08-09T14:51:47.932 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/locale 2024-08-09T14:51:47.932 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.932 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG_backup=en_US.UTF-8 2024-08-09T14:51:47.933 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.933 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /etc/locale.conf ']' 2024-08-09T14:51:47.933 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -x /usr/bin/sed ']' 2024-08-09T14:51:47.933 INFO:tasks.workunit.client.0.smithi155.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-08-09T14:51:47.933 INFO:tasks.workunit.client.0.smithi155.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-08-09T14:51:47.933 INFO:tasks.workunit.client.0.smithi155.stderr:++++ export LANG=en_US.UTF-8 2024-08-09T14:51:47.933 INFO:tasks.workunit.client.0.smithi155.stderr:++++ LANG=en_US.UTF-8 2024-08-09T14:51:47.933 INFO:tasks.workunit.client.0.smithi155.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-08-09T14:51:47.933 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-08-09T14:51:47.933 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.933 INFO:tasks.workunit.client.0.smithi155.stderr:+++ LANG=en_US.UTF-8 2024-08-09T14:51:47.933 INFO:tasks.workunit.client.0.smithi155.stderr:+++ unset LANG_backup config 2024-08-09T14:51:47.933 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n '' ']' 2024-08-09T14:51:47.934 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-08-09T14:51:47.934 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' dumb = linux ']' 2024-08-09T14:51:47.934 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.934 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-08-09T14:51:47.934 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.934 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/less.sh 2024-08-09T14:51:47.934 INFO:tasks.workunit.client.0.smithi155.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-08-09T14:51:47.934 INFO:tasks.workunit.client.0.smithi155.stderr:++ for i in /etc/profile.d/*.sh 2024-08-09T14:51:47.934 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-08-09T14:51:47.934 INFO:tasks.workunit.client.0.smithi155.stderr:++ '[' '' ']' 2024-08-09T14:51:47.934 INFO:tasks.workunit.client.0.smithi155.stderr:++ . /etc/profile.d/which2.sh 2024-08-09T14:51:47.934 INFO:tasks.workunit.client.0.smithi155.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-08-09T14:51:47.934 INFO:tasks.workunit.client.0.smithi155.stderr:+++++ readlink /proc/42587/exe 2024-08-09T14:51:47.935 INFO:tasks.workunit.client.0.smithi155.stderr:++++ basename /usr/bin/bash 2024-08-09T14:51:47.935 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_declare='declare -f' 2024-08-09T14:51:47.935 INFO:tasks.workunit.client.0.smithi155.stderr:+++ which_opt=-f 2024-08-09T14:51:47.935 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export which_declare 2024-08-09T14:51:47.935 INFO:tasks.workunit.client.0.smithi155.stderr:+++ export -f which 2024-08-09T14:51:47.935 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset i 2024-08-09T14:51:47.935 INFO:tasks.workunit.client.0.smithi155.stderr:++ unset -f pathmunge 2024-08-09T14:51:47.935 INFO:tasks.workunit.client.0.smithi155.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-08-09T14:51:47.935 INFO:tasks.workunit.client.0.smithi155.stderr:+ export PATH 2024-08-09T14:51:47.935 INFO:tasks.workunit.client.0.smithi155.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-08-09T14:51:47.935 INFO:tasks.workunit.client.0.smithi155.stderr:+ unset rc 2024-08-09T14:51:47.935 INFO:tasks.workunit.client.0.smithi155.stderr:+ ceph_test_neorados_write_operations 2024-08-09T14:51:47.935 INFO:tasks.workunit.client.0.smithi155.stderr:+ sed 's/^/ write_operations: /' 2024-08-09T14:51:47.936 INFO:tasks.workunit.client.0.smithi155.stderr:+ tee ceph_test_neorados_write_operations.log 2024-08-09T14:51:50.057 INFO:tasks.workunit.client.0.smithi155.stdout:=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:51:50.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.400+0000 7fc3c91c2640 1 --2- 172.21.15.155:0/3100773278 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fc3ac06ba30 0x7fc3ac06def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:51:50.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.401+0000 7fc3c91c2640 10 monclient: get_auth_request con 0x7fc3ac06ba30 auth_method 0 2024-08-09T14:51:50.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.401+0000 7fc3c91c2640 1 --2- 172.21.15.155:0/3100773278 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fc3ac06ba30 0x7fc3ac06def0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4055000 tx=0x7fc3c404f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:51:50.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.401+0000 7fc3ba7fc640 10 client.4435.objecter ms_handle_connect 0x7fc3ac06ba30 2024-08-09T14:51:50.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.405+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 5238+0+0 (secure 0 0 0) 0x7fc3bc0577b0 con 0x55a86d4642e0 2024-08-09T14:51:50.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.405+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(28..28 src has 1..28) 2024-08-09T14:51:50.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [28,28] > 0 2024-08-09T14:51:50.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding full epoch 28 2024-08-09T14:51:50.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.405+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:51:50.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.405+0000 7fc3cb6c6780 10 librados: wait_for_osdmap done waiting 2024-08-09T14:51:50.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.405+0000 7fc3cb6c6780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:51:50.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:47.405+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc3bc00f400 con 0x55a86d4642e0 2024-08-09T14:51:50.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:48.117+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fc3bc02a020 con 0x55a86d4642e0 2024-08-09T14:51:50.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:48.117+0000 7fc3ba7fc640 10 monclient: handle_get_version_reply finishing 1 version 28 2024-08-09T14:51:50.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:48.117+0000 7fc3baffd640 10 client.4435.objecter _get_latest_version latest 28, have it 2024-08-09T14:51:50.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:48.117+0000 7fc3cb6c6780 10 client.4435.objecter create_pool name=ceph_test_rados_api_asio 2024-08-09T14:51:50.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:48.117+0000 7fc3cb6c6780 10 client.4435.objecter pool_op_submit 1 2024-08-09T14:51:50.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:48.117+0000 7fc3cb6c6780 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:51:50.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:48.117+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x55a86d3df800 con 0x55a86d4642e0 2024-08-09T14:51:50.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.055+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v30) ==== 43+0+0 (secure 0 0 0) 0x55a86d4f42d0 con 0x55a86d4642e0 2024-08-09T14:51:50.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.055+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 pool_op_reply(tid 1 (0) Success v30) 2024-08-09T14:51:50.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.055+0000 7fc3ba7fc640 10 client.4435.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v30) 2024-08-09T14:51:50.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.055+0000 7fc3ba7fc640 10 client.4435.objecter have request 1 at 0x55a86d4f7ab0 Op: create 2024-08-09T14:51:50.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.055+0000 7fc3ba7fc640 20 client.4435.objecter waiting for client to reach epoch 30 before calling back 2024-08-09T14:51:50.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.055+0000 7fc3ba7fc640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:51:50.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.055+0000 7fc3ba7fc640 10 monclient: _renew_subs 2024-08-09T14:51:50.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.055+0000 7fc3ba7fc640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:51:50.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.055+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=29}) -- 0x7fc3ac0717a0 con 0x55a86d4642e0 2024-08-09T14:51:50.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.055+0000 7fc3ba7fc640 10 client.4435.objecter done 2024-08-09T14:51:50.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.330+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:51:50.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.330+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:51:50.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.330+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:51:50.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.330+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:51:50.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.348+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 8 ==== osd_map(29..30 src has 1..30) ==== 15556+0+0 (secure 0 0 0) 0x7fc3bc057a10 con 0x55a86d4642e0 2024-08-09T14:51:50.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.348+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(29..30 src has 1..30) 2024-08-09T14:51:50.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [29,30] > 28 2024-08-09T14:51:50.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 29 2024-08-09T14:51:50.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 30 2024-08-09T14:51:50.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:51:50.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 10 librados: set snap read head -> 1 2024-08-09T14:51:50.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 10 librados: writefull oid=exist nspace= 2024-08-09T14:51:50.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d4e3b90 2024-08-09T14:51:50.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 30 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:51:50.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:51:50.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:51:50.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 1 --2- 172.21.15.155:0/3100773278 >> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] conn(0x55a86d4e4550 0x55a86d5a8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T14:51:50.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:51:50.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid exist '@20' '@20' [writefull 0~5 in=5b] tid 2 osd.0 2024-08-09T14:51:50.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:51:50.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 2 2024-08-09T14:51:50.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 2 to 20.1 on osd.0 2024-08-09T14:51:59.131 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:2 20.1 20:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) -- 0x55a86d4e4980 con 0x55a86d4e4550 2024-08-09T14:51:59.131 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:51:59.131 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:50.349+0000 7fc3cb6c6780 5 client.4435.objecter 1 in flight 2024-08-09T14:51:59.131 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:51.330+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:51:59.131 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:51.330+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:51:59.131 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:51.330+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:51:59.131 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:52.330+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:51:59.131 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:52.330+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:51:59.131 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:52.330+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:51:59.131 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:52.367+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:51:59.132 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:53.331+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:51:59.132 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:53.331+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:51:59.132 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:53.331+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:51:59.132 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:54.331+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:51:59.132 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:54.331+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:51:59.132 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:54.331+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:51:59.132 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:55.331+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:51:59.132 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:55.331+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:51:59.132 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:55.331+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:51:59.132 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:56.331+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:51:59.132 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:56.331+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:51:59.132 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:56.331+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:51:59.133 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:57.331+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:51:59.133 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:57.331+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:51:59.133 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:57.331+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:51:59.133 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:57.368+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:51:59.133 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:58.332+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:51:59.133 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:58.332+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:51:59.133 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:58.332+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:51:59.133 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:59.125+0000 7fc3c99c3640 1 --2- 172.21.15.155:0/3100773278 >> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] conn(0x55a86d4e4550 0x55a86d5a8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-08-09T14:51:59.133 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:59.126+0000 7fc3c99c3640 10 monclient: get_auth_request con 0x55a86d4e4550 auth_method 0 2024-08-09T14:52:00.298 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:35.542 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:00.312 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:02:35.559 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:00.312 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:35.556 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:00.319 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:02:35.566 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:00.333 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:59.129+0000 7fc3c99c3640 1 --2- 172.21.15.155:0/3100773278 >> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] conn(0x55a86d4e4550 0x55a86d5a8a00 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-08-09T14:52:00.333 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:59.129+0000 7fc3ba7fc640 10 client.4435.objecter ms_handle_connect 0x55a86d4e4550 2024-08-09T14:52:00.333 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:59.332+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:00.333 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:59.332+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:00.333 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:51:59.332+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:00.334 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 1 ==== osd_map(31..35 src has 1..35) ==== 15256+0+0 (crc 0 0 0) 0x7fc3c0005b20 con 0x55a86d4e4550 2024-08-09T14:52:00.334 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(31..35 src has 1..35) 2024-08-09T14:52:00.334 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [31,35] > 30 2024-08-09T14:52:00.334 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 31 2024-08-09T14:52:00.334 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:00.334 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 31 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:00.334 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:00.334 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 32 2024-08-09T14:52:00.334 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:00.334 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 32 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:00.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:00.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 33 2024-08-09T14:52:00.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:00.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 33 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:00.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:00.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 34 2024-08-09T14:52:00.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:00.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 34 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:00.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:00.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 35 2024-08-09T14:52:00.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:00.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 35 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:00.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:00.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:52:00.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.317+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:52:00.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.332+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:00.399 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:35.642 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:00.410 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:35.653 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:00.451 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:02:35.698 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:00.597 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:02:35.840 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:00.673 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:02:35.917 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:02.644 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.332+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:02.644 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.332+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:02.644 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:00.332+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:52:02.644 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:01.332+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:02.644 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:01.332+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:02.645 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:01.332+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:02.645 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.332+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:02.645 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.332+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:02.645 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.332+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:02.645 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.369+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:52:02.645 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.369+0000 7fc3cb426640 2 client.4435.objecter tid 2 on osd.0 is laggy 2024-08-09T14:52:02.645 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.369+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:52:02.645 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.369+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:52:02.645 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.369+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:52:02.645 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.369+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=36}) -- 0x7fc39c002f90 con 0x55a86d4642e0 2024-08-09T14:52:02.645 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.369+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c003360 con 0x55a86d4e4550 2024-08-09T14:52:02.646 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 9 ==== osd_map(36..39 src has 1..39) ==== 5950+0+0 (secure 0 0 0) 0x7fc3bc0596a0 con 0x55a86d4642e0 2024-08-09T14:52:02.646 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(36..39 src has 1..39) 2024-08-09T14:52:02.646 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [36,39] > 35 2024-08-09T14:52:02.646 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 36 2024-08-09T14:52:02.646 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:02.646 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 36 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:02.646 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:02.646 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 37 2024-08-09T14:52:02.646 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:02.646 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 37 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:02.646 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:02.646 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 38 2024-08-09T14:52:02.647 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:02.647 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 38 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:07.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:07.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 39 2024-08-09T14:52:07.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:07.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 39 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:07.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:07.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:52:07.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:02.642+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:52:07.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:03.332+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:07.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:03.332+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:07.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:03.332+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:07.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:04.332+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:07.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:04.332+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:07.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:04.332+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:07.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:05.332+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:07.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:05.332+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:07.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:05.332+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:07.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:06.332+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:07.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:06.332+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:07.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:06.332+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:07.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.332+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:07.475 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.332+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:07.475 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.332+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:07.475 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.370+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:52:07.475 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.370+0000 7fc3cb426640 2 client.4435.objecter tid 2 on osd.0 is laggy 2024-08-09T14:52:07.475 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.370+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:52:07.475 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.370+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:52:07.475 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.370+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:52:07.475 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.370+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=40}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:52:07.475 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.370+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c003840 con 0x55a86d4e4550 2024-08-09T14:52:07.475 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 10 ==== osd_map(40..42 src has 1..42) ==== 2273+0+0 (secure 0 0 0) 0x7fc3bc062200 con 0x55a86d4642e0 2024-08-09T14:52:07.475 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(40..42 src has 1..42) 2024-08-09T14:52:12.372 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [40,42] > 39 2024-08-09T14:52:12.372 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 40 2024-08-09T14:52:12.373 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:12.373 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 40 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:12.373 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:12.373 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 41 2024-08-09T14:52:12.373 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:12.373 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 41 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:12.373 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:12.373 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 42 2024-08-09T14:52:12.373 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:12.373 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 42 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:12.373 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:12.373 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:52:12.374 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:07.471+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:52:12.374 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:08.332+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:12.374 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:08.332+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:12.374 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:08.332+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:12.374 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:09.333+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:12.374 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:09.333+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:12.374 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:09.333+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:12.374 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:10.333+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:12.374 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:10.333+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:12.374 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:10.333+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:12.374 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:10.333+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:52:12.374 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:11.333+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:12.374 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:11.333+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:12.374 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:11.333+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:12.375 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.333+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:12.375 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.333+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:12.375 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.333+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:12.375 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.371+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:52:12.375 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.371+0000 7fc3cb426640 2 client.4435.objecter tid 2 on osd.0 is laggy 2024-08-09T14:52:12.375 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.371+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:52:13.836 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.371+0000 7fc3cb426 pool: Running main() from gmock_main.cc 2024-08-09T14:52:13.836 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-08-09T14:52:13.836 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [----------] Global test environment set-up. 2024-08-09T14:52:13.836 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-08-09T14:52:13.836 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-08-09T14:52:13.836 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [ OK ] NeoRadosPools.PoolList (3872 ms) 2024-08-09T14:52:13.836 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-08-09T14:52:13.836 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (3062 ms) 2024-08-09T14:52:13.836 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-08-09T14:52:13.836 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (4018 ms) 2024-08-09T14:52:13.837 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-08-09T14:52:13.837 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (9210 ms) 2024-08-09T14:52:13.837 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-08-09T14:52:13.837 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2854 ms) 2024-08-09T14:52:13.837 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-08-09T14:52:13.837 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (3115 ms) 2024-08-09T14:52:13.837 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [----------] 6 tests from NeoRadosPools (26131 ms total) 2024-08-09T14:52:13.837 INFO:tasks.workunit.client.0.smithi155.stdout: pool: 2024-08-09T14:52:13.837 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [----------] Global test environment tear-down 2024-08-09T14:52:13.837 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [==========] 6 tests from 1 test suite ran. (26131 ms total) 2024-08-09T14:52:13.837 INFO:tasks.workunit.client.0.smithi155.stdout: pool: [ PASSED ] 6 tests. 2024-08-09T14:52:16.335 INFO:tasks.workunit.client.0.smithi155.stdout:640 10 monclient: _renew_subs 2024-08-09T14:52:16.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.371+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:52:16.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.371+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=43}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:52:16.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.371+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c003aa0 con 0x55a86d4e4550 2024-08-09T14:52:16.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.499+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 11 ==== osd_map(43..45 src has 1..45) ==== 2992+0+0 (secure 0 0 0) 0x7fc3bc055d80 con 0x55a86d4642e0 2024-08-09T14:52:16.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.499+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(43..45 src has 1..45) 2024-08-09T14:52:16.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.499+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [43,45] > 42 2024-08-09T14:52:16.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.499+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 43 2024-08-09T14:52:16.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.500+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:16.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.500+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 43 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:16.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.500+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:16.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.500+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 44 2024-08-09T14:52:16.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.500+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:16.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.500+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 44 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:16.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.500+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:16.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.500+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 45 2024-08-09T14:52:16.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.500+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:16.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.500+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 45 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:16.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.500+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:16.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.500+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:52:16.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:12.500+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:52:16.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:13.333+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:16.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:13.333+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:16.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:13.333+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:16.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:14.333+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:16.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:14.333+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:16.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:14.333+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:16.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:15.333+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:16.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:15.333+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:16.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:15.333+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:17.835 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:16.334+0000 7fc3b97fa640 10 monclie api_service: [==========] Running 4 tests from 1 test suite. 2024-08-09T14:52:17.835 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: [----------] Global test environment set-up. 2024-08-09T14:52:17.835 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: [----------] 4 tests from LibRadosService 2024-08-09T14:52:17.835 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-08-09T14:52:17.835 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5685 ms) 2024-08-09T14:52:17.835 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-08-09T14:52:17.835 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: [ OK ] LibRadosService.RegisterLate (146 ms) 2024-08-09T14:52:17.835 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-08-09T14:52:17.835 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: cluster: 2024-08-09T14:52:17.835 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: id: f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:52:17.835 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: health: HEALTH_WARN 2024-08-09T14:52:17.836 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 20 pool(s) do not have an application enabled 2024-08-09T14:52:17.836 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 2024-08-09T14:52:17.836 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: services: 2024-08-09T14:52:17.836 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-08-09T14:52:17.836 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-08-09T14:52:17.836 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: osd: 8 osds: 8 up (since 75s), 8 in (since 2m) 2024-08-09T14:52:17.836 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-08-09T14:52:17.836 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 2024-08-09T14:52:17.836 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: data: 2024-08-09T14:52:17.836 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: pools: 35 pools, 1065 pgs 2024-08-09T14:52:17.836 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: objects: 4 objects, 449 KiB 2024-08-09T14:52:17.836 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-08-09T14:52:17.837 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: pgs: 94.836% pgs unknown 2024-08-09T14:52:17.837 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 4.319% pgs not active 2024-08-09T14:52:17.837 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 1010 unknown 2024-08-09T14:52:17.837 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 43 creating+peering 2024-08-09T14:52:17.837 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 9 active+clean 2024-08-09T14:52:17.837 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 3 creating+activating 2024-08-09T14:52:17.837 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 2024-08-09T14:52:17.837 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 2024-08-09T14:52:17.837 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: cluster: 2024-08-09T14:52:17.837 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: id: f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T14:52:17.837 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: health: HEALTH_WARN 2024-08-09T14:52:17.837 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 15 pool(s) do not have an application enabled 2024-08-09T14:52:17.837 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 2024-08-09T14:52:17.837 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: services: 2024-08-09T14:52:17.838 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-08-09T14:52:17.838 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-08-09T14:52:17.838 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: osd: 8 osds: 8 up (since 78s), 8 in (since 2m) 2024-08-09T14:52:17.838 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-08-09T14:52:17.838 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-08-09T14:52:17.838 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 2024-08-09T14:52:17.838 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: data: 2024-08-09T14:52:17.838 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: pools: 35 pools, 1017 pgs 2024-08-09T14:52:17.838 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: objects: 4 objects, 449 KiB 2024-08-09T14:52:17.838 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: usage: 215 MiB used, 720 GiB / 720 GiB avail 2024-08-09T14:52:17.838 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: pgs: 20.944% pgs unknown 2024-08-09T14:52:17.838 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 75.418% pgs not active 2024-08-09T14:52:17.839 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 736 creating+peering 2024-08-09T14:52:17.839 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 213 unknown 2024-08-09T14:52:17.839 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 37 active+clean 2024-08-09T14:52:17.839 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 31 creating+activating 2024-08-09T14:52:17.839 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 2024-08-09T14:52:17.839 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 2024-08-09T14:52:17.839 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4458 ms) 2024-08-09T14:52:17.839 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: [ RUN ] LibRadosService.Status 2024-08-09T14:52:17.839 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: [ OK ] LibRadosService.Status (20058 ms) 2024-08-09T14:52:17.839 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: [----------] 4 tests from LibRadosService (30348 ms total) 2024-08-09T14:52:17.839 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: 2024-08-09T14:52:17.839 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: [----------] Global test environment tear-down 2024-08-09T14:52:17.840 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: [==========] 4 tests from 1 test suite ran. (30350 ms total) 2024-08-09T14:52:17.840 INFO:tasks.workunit.client.0.smithi155.stdout: api_service: [ PASSED ] 4 tests. 2024-08-09T14:52:18.335 INFO:tasks.workunit.client.0.smithi155.stdout:nt: tick 2024-08-09T14:52:18.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:16.334+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:18.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:16.334+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:18.335 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.334+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:18.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.334+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:18.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.334+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:18.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.372+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:52:18.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.372+0000 7fc3cb426640 2 client.4435.objecter tid 2 on osd.0 is laggy 2024-08-09T14:52:18.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.372+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:52:18.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.372+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:52:18.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.372+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:52:18.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.372+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=46}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:52:18.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.372+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c003cf0 con 0x55a86d4e4550 2024-08-09T14:52:18.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.644+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 12 ==== osd_map(46..48 src has 1..48) ==== 1609+0+0 (secure 0 0 0) 0x7fc3bc055d80 con 0x55a86d4642e0 2024-08-09T14:52:18.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(46..48 src has 1..48) 2024-08-09T14:52:18.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [46,48] > 45 2024-08-09T14:52:18.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 46 2024-08-09T14:52:18.336 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:18.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 46 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:18.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:18.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 47 2024-08-09T14:52:18.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:18.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 47 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:18.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:18.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 48 2024-08-09T14:52:18.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:18.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 48 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:18.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:18.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:52:18.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:17.645+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:52:19.092 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:18.33 api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-08-09T14:52:19.092 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: [----------] Global test environment set-up. 2024-08-09T14:52:19.092 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-08-09T14:52:19.093 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-08-09T14:52:19.093 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5505 ms) 2024-08-09T14:52:19.093 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-08-09T14:52:19.093 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (269 ms) 2024-08-09T14:52:19.093 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-08-09T14:52:19.093 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20249 ms) 2024-08-09T14:52:19.093 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-08-09T14:52:19.093 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: attempt 0 of 20 2024-08-09T14:52:19.093 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: failed to deregister: 2024-08-09T14:52:19.093 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: { 2024-08-09T14:52:19.093 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "epoch": 12, 2024-08-09T14:52:19.093 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "modified": "2024-08-09T14:52:15.588617+0000", 2024-08-09T14:52:19.093 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "services": { 2024-08-09T14:52:19.094 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "laundry": { 2024-08-09T14:52:19.094 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "daemons": { 2024-08-09T14:52:19.094 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "summary": "", 2024-08-09T14:52:19.094 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "close-test-pid42105": { 2024-08-09T14:52:19.094 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "start_epoch": 12, 2024-08-09T14:52:19.094 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "start_stamp": "2024-08-09T14:52:13.993897+0000", 2024-08-09T14:52:19.094 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "gid": 4699, 2024-08-09T14:52:19.094 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "addr": "172.21.15.155:0/2442943722", 2024-08-09T14:52:19.094 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "metadata": { 2024-08-09T14:52:19.094 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "arch": "x86_64", 2024-08-09T14:52:19.095 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "ceph_release": "squid", 2024-08-09T14:52:19.095 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "ceph_version": "ceph version 19.1.0-1345-g0a88632b (0a88632b45f25ae38493667c7dbd155f6e59b499) squid (rc)", 2024-08-09T14:52:19.095 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "ceph_version_short": "19.1.0-1345-g0a88632b", 2024-08-09T14:52:19.095 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-09T14:52:19.095 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "distro": "centos", 2024-08-09T14:52:19.095 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-08-09T14:52:19.096 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "distro_version": "9", 2024-08-09T14:52:19.096 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "foo": "bar", 2024-08-09T14:52:19.096 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "hostname": "smithi155", 2024-08-09T14:52:19.096 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024", 2024-08-09T14:52:19.096 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "kernel_version": "5.14.0-480.el9.x86_64", 2024-08-09T14:52:19.096 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "mem_swap_kb": "0", 2024-08-09T14:52:19.096 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "mem_total_kb": "32487388", 2024-08-09T14:52:19.096 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "os": "Linux", 2024-08-09T14:52:19.097 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "this": "that" 2024-08-09T14:52:19.097 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: }, 2024-08-09T14:52:19.097 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "task_status": {} 2024-08-09T14:52:19.097 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: }, 2024-08-09T14:52:19.097 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "pid42036": { 2024-08-09T14:52:19.097 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "start_epoch": 10, 2024-08-09T14:52:19.097 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "start_stamp": "2024-08-09T14:51:57.852710+0000", 2024-08-09T14:52:19.097 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "gid": 4586, 2024-08-09T14:52:19.097 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "addr": "172.21.15.155:0/142538250", 2024-08-09T14:52:19.097 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "metadata": { 2024-08-09T14:52:19.097 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "arch": "x86_64", 2024-08-09T14:52:19.097 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "ceph_release": "squid", 2024-08-09T14:52:19.098 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "ceph_version": "ceph version 19.1.0-1345-g0a88632b (0a88632b45f25ae38493667c7dbd155f6e59b499) squid (rc)", 2024-08-09T14:52:19.098 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "ceph_version_short": "19.1.0-1345-g0a88632b", 2024-08-09T14:52:19.098 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-09T14:52:22.418 INFO:tasks.workunit.client.0.smithi155.stdout: 4+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:22.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:18.334+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:22.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:18.334+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:22.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:19.334+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:22.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:19.334+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:22.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:19.334+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:22.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:20.334+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:22.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:20.334+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:22.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:20.334+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:22.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:20.334+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:52:22.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:21.334+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:22.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:21.334+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:22.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:21.334+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:22.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.334+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:22.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.334+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:22.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.334+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:22.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.373+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:52:22.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.373+0000 7fc3cb426640 2 client.4435.objecter tid 2 on osd.0 is laggy 2024-08-09T14:52:22.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.373+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:52:22.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.373+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:52:22.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.373+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:52:22.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.373+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=49}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:52:22.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.373+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c003f50 con 0x55a86d4e4550 2024-08-09T14:52:22.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.416+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 13 ==== osd_map(49..50 src has 1..50) ==== 1291+0+0 (secure 0 0 0) 0x7fc3bc055d80 con 0x55a86d4642e0 2024-08-09T14:52:22.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.416+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(49..50 src has 1..50) 2024-08-09T14:52:22.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.416+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [49,50] > 48 2024-08-09T14:52:22.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.416+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 49 2024-08-09T14:52:22.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.416+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:22.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.416+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 49 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:22.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.416+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:22.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.416+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 50 2024-08-09T14:52:22.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.416+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:22.728 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:22.522+0000 ea5d640 -1 osd.1 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:23.748 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:23.654+0000 ea5d640 -1 osd.1 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:23.960 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:23.919+0000 12a65640 -1 osd.0 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:24.589 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52: handler_error: Running main() from gmock_main.cc 2024-08-09T14:52:24.589 INFO:tasks.workunit.client.0.smithi155.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-08-09T14:52:24.589 INFO:tasks.workunit.client.0.smithi155.stdout: handler_error: [----------] Global test environment set-up. 2024-08-09T14:52:24.589 INFO:tasks.workunit.client.0.smithi155.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-08-09T14:52:24.589 INFO:tasks.workunit.client.0.smithi155.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-08-09T14:52:24.589 INFO:tasks.workunit.client.0.smithi155.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (36956 ms) 2024-08-09T14:52:24.589 INFO:tasks.workunit.client.0.smithi155.stdout: handler_error: [----------] 1 test from neocls_handler_error (36956 ms total) 2024-08-09T14:52:24.589 INFO:tasks.workunit.client.0.smithi155.stdout: handler_error: 2024-08-09T14:52:24.589 INFO:tasks.workunit.client.0.smithi155.stdout: handler_error: [----------] Global test environment tear-down 2024-08-09T14:52:24.589 INFO:tasks.workunit.client.0.smithi155.stdout: handler_error: [==========] 1 test from 1 test suite ran. (36956 ms total) 2024-08-09T14:52:24.590 INFO:tasks.workunit.client.0.smithi155.stdout: handler_error: [ PASSED ] 1 test. 2024-08-09T14:52:24.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:24.684+0000 ea5d640 -1 osd.1 50 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:25.088 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:25.077+0000 12a65640 -1 osd.0 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:25.952 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:01.195 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:26.030 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:01.273 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:26.076 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:01.319 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:26.107 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:01.354 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:26.112 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:01.359 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:26.118 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:01.362 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:26.175 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:26.116+0000 ea5d640 -1 osd.1 51 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:26.309 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:01.556 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:26.352 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:01.595 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:26.365 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:52:26.210+0000 ea5d640 -1 osd.4 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.6s0 39.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-08-09T14:52:26.436 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:01.679 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:26.725 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:26.572+0000 12a65640 -1 osd.0 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:27.378 INFO:tasks.workunit.client.0.smithi155.stdout:22.416+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 50 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:27.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.416+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:27.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.416+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:52:27.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:22.416+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:52:27.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:23.334+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:27.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:23.334+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:27.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:23.334+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:27.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:24.335+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:27.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:24.335+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:27.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:24.335+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:27.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:25.335+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:27.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:25.335+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:27.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:25.335+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:27.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:26.335+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:27.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:26.335+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:27.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:26.335+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:27.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.335+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:27.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.335+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:27.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.335+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:27.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.374+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:52:27.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.374+0000 7fc3cb426640 2 client.4435.objecter tid 2 on osd.0 is laggy 2024-08-09T14:52:27.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.374+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:52:27.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.374+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:52:27.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.374+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:52:27.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.374+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=51}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:52:27.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.374+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004200 con 0x55a86d4e4550 2024-08-09T14:52:27.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 14 ==== osd_map(51..54 src has 1..54) ==== 4845+0+0 (secure 0 0 0) 0x7fc3bc079e00 con 0x55a86d4642e0 2024-08-09T14:52:27.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(51..54 src has 1..54) 2024-08-09T14:52:27.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [51,54] > 50 2024-08-09T14:52:27.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 51 2024-08-09T14:52:27.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:27.948 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:52:27.899+0000 ea5d640 -1 osd.4 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.6s0 39.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-08-09T14:52:28.523 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:28.386+0000 12a65640 -1 osd.0 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:29.424 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:29.405+0000 12a65640 -1 osd.0 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:29.500 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:52:29.462+0000 ea5d640 -1 osd.4 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.6s0 39.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-08-09T14:52:30.054 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 51 base exist @20 precalc_pgid 0 pgid 0. api_service_pp: "distro": "centos", 2024-08-09T14:52:30.054 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-08-09T14:52:30.054 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "distro_version": "9", 2024-08-09T14:52:30.054 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "foo": "bar", 2024-08-09T14:52:30.054 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "hostname": "smithi155", 2024-08-09T14:52:30.055 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024", 2024-08-09T14:52:30.055 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "kernel_version": "5.14.0-480.el9.x86_64", 2024-08-09T14:52:30.055 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "mem_swap_kb": "0", 2024-08-09T14:52:30.055 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "mem_total_kb": "32487388", 2024-08-09T14:52:30.055 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "os": "Linux", 2024-08-09T14:52:30.055 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "this": "that" 2024-08-09T14:52:30.055 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: }, 2024-08-09T14:52:30.055 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "task_status": {} 2024-08-09T14:52:30.055 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:30.055 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:30.055 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:30.055 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:30.056 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:30.056 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: 2024-08-09T14:52:30.056 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: attempt 1 of 20 2024-08-09T14:52:30.056 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: failed to deregister: 2024-08-09T14:52:30.056 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: { 2024-08-09T14:52:30.056 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "epoch": 14, 2024-08-09T14:52:30.056 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "modified": "2024-08-09T14:52:19.590164+0000", 2024-08-09T14:52:30.056 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "services": { 2024-08-09T14:52:30.056 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "laundry": { 2024-08-09T14:52:30.056 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "daemons": { 2024-08-09T14:52:30.056 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "summary": "", 2024-08-09T14:52:30.056 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "close-test-pid42105": { 2024-08-09T14:52:30.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "start_epoch": 14, 2024-08-09T14:52:30.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "start_stamp": "2024-08-09T14:52:19.492810+0000", 2024-08-09T14:52:30.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "gid": 4634, 2024-08-09T14:52:30.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "addr": "172.21.15.155:0/560967942", 2024-08-09T14:52:30.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "metadata": { 2024-08-09T14:52:30.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "arch": "x86_64", 2024-08-09T14:52:30.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "ceph_release": "squid", 2024-08-09T14:52:30.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "ceph_version": "ceph version 19.1.0-1345-g0a88632b (0a88632b45f25ae38493667c7dbd155f6e59b499) squid (rc)", 2024-08-09T14:52:30.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "ceph_version_short": "19.1.0-1345-g0a88632b", 2024-08-09T14:52:30.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-09T14:52:30.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "distro": "centos", 2024-08-09T14:52:30.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-08-09T14:52:30.057 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "distro_version": "9", 2024-08-09T14:52:30.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "foo": "bar", 2024-08-09T14:52:30.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "hostname": "smithi155", 2024-08-09T14:52:30.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024", 2024-08-09T14:52:30.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "kernel_version": "5.14.0-480.el9.x86_64", 2024-08-09T14:52:30.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "mem_swap_kb": "0", 2024-08-09T14:52:30.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "mem_total_kb": "32487388", 2024-08-09T14:52:30.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "os": "Linux", 2024-08-09T14:52:30.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "this": "that" 2024-08-09T14:52:30.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: }, 2024-08-09T14:52:30.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "task_status": {} 2024-08-09T14:52:30.058 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:30.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:30.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:30.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:30.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:30.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: 2024-08-09T14:52:30.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: attempt 2 of 20 2024-08-09T14:52:30.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: failed to deregister: 2024-08-09T14:52:30.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: { 2024-08-09T14:52:30.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "epoch": 16, 2024-08-09T14:52:30.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "modified": "2024-08-09T14:52:25.594104+0000", 2024-08-09T14:52:30.059 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "services": { 2024-08-09T14:52:30.449 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:30.431+0000 12a65640 -1 osd.0 54 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:31.073 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:52:30.962+0000 ea5d640 -1 osd.4 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.6s0 39.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-08-09T14:52:31.533 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:31.518+0000 12a65640 -1 osd.0 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:32.130 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:52:32.069+0000 ea5d640 -1 osd.4 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.6s0 39.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-08-09T14:52:32.376 INFO:tasks.workunit.client.0.smithi155.stdout: 0 is_write 2024-08-09T14:52:32.376 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:32.376 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 52 2024-08-09T14:52:32.376 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:32.376 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 52 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:32.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:32.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 53 2024-08-09T14:52:32.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:32.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 53 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:32.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:32.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 54 2024-08-09T14:52:32.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:32.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 54 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:32.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:32.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:52:32.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:27.377+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:52:32.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:28.335+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:32.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:28.335+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:32.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:28.335+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:32.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:29.335+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:32.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:29.335+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:32.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:29.335+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:32.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:30.335+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:32.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:30.335+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:32.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:30.335+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:32.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:30.335+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:52:32.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:31.335+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:32.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:31.335+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:32.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:31.335+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:32.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.335+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:32.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.335+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:32.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.335+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:32.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.375+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:52:32.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.375+0000 7fc3cb426640 2 client.4435.objecter tid 2 on osd.0 is laggy 2024-08-09T14:52:32.682 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:32.655+0000 12a65640 -1 osd.0 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:33.436 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:52:33.354+0000 ea5d640 -1 osd.4 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.6s0 39.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-08-09T14:52:33.847 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:33.842+0000 12a65640 -1 osd.0 57 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:34.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.375+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:52:34.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.375+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:52:34.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.375+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:52:34.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.375+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=55}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:52:34.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.375+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004480 con 0x55a86d4e4550 2024-08-09T14:52:34.337 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.378+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 15 ==== osd_map(55..58 src has 1..58) ==== 2540+0+0 (secure 0 0 0) 0x7fc3bc059440 con 0x55a86d4642e0 2024-08-09T14:52:34.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.378+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(55..58 src has 1..58) 2024-08-09T14:52:34.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.378+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [55,58] > 54 2024-08-09T14:52:34.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.378+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 55 2024-08-09T14:52:34.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.378+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:34.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.378+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 55 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:34.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.378+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:34.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.378+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 56 2024-08-09T14:52:34.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.378+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:34.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.378+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 56 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:34.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.378+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:34.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.378+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 57 2024-08-09T14:52:34.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.379+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:34.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.379+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 57 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:34.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.379+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:34.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.379+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 58 2024-08-09T14:52:34.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.379+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:34.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.379+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 58 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:34.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.379+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:34.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.379+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:52:34.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:32.379+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:52:34.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:33.335+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:34.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:33.335+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:34.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:33.335+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:35.055 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:35.046+0000 12a65640 -1 osd.0 58 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:35.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 20 api_service_pp: "laundry": { 2024-08-09T14:52:35.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "daemons": { 2024-08-09T14:52:35.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "summary": "", 2024-08-09T14:52:35.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "close-test-pid42105": { 2024-08-09T14:52:35.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "start_epoch": 16, 2024-08-09T14:52:35.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "start_stamp": "2024-08-09T14:52:24.987305+0000", 2024-08-09T14:52:35.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "gid": 4729, 2024-08-09T14:52:35.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "addr": "172.21.15.155:0/3264875288", 2024-08-09T14:52:35.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "metadata": { 2024-08-09T14:52:35.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "arch": "x86_64", 2024-08-09T14:52:35.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "ceph_release": "squid", 2024-08-09T14:52:35.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "ceph_version": "ceph version 19.1.0-1345-g0a88632b (0a88632b45f25ae38493667c7dbd155f6e59b499) squid (rc)", 2024-08-09T14:52:35.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "ceph_version_short": "19.1.0-1345-g0a88632b", 2024-08-09T14:52:35.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-09T14:52:35.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "distro": "centos", 2024-08-09T14:52:35.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-08-09T14:52:35.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "distro_version": "9", 2024-08-09T14:52:35.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "foo": "bar", 2024-08-09T14:52:35.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "hostname": "smithi155", 2024-08-09T14:52:35.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Fri Jul 12 20:45:27 UTC 2024", 2024-08-09T14:52:35.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "kernel_version": "5.14.0-480.el9.x86_64", 2024-08-09T14:52:35.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "mem_swap_kb": "0", 2024-08-09T14:52:35.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "mem_total_kb": "32487388", 2024-08-09T14:52:35.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "os": "Linux", 2024-08-09T14:52:35.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "this": "that" 2024-08-09T14:52:35.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: }, 2024-08-09T14:52:35.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: "task_status": {} 2024-08-09T14:52:35.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:35.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:35.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:35.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:35.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: } 2024-08-09T14:52:35.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: 2024-08-09T14:52:35.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: attempt 3 of 20 2024-08-09T14:52:35.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (21965 ms) 2024-08-09T14:52:35.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (47988 ms total) 2024-08-09T14:52:35.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: 2024-08-09T14:52:35.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: [----------] Global test environment tear-down 2024-08-09T14:52:35.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (47989 ms total) 2024-08-09T14:52:35.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-08-09T14:52:35.603 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:52:35.555+0000 ea5d640 -1 osd.4 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.6s0 39.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-08-09T14:52:36.158 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:36.081+0000 12a65640 -1 osd.0 59 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:36.696 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:52:36.678+0000 ea5d640 -1 osd.4 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.6s0 39.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-08-09T14:52:37.428 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:37.348+0000 12a65640 -1 osd.0 60 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:37.598 INFO:tasks.workunit.client.0.smithi155.stdout:24-08-09T14:52:34.335+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:37.599 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:34.335+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:37.599 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:34.335+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:37.599 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:35.336+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:37.599 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:35.336+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:37.599 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:35.336+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:37.599 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:36.336+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:37.599 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:36.336+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:37.599 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:36.336+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:37.599 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.336+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:37.599 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.336+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:37.599 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.336+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:37.599 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.375+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:52:37.600 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.375+0000 7fc3cb426640 2 client.4435.objecter tid 2 on osd.0 is laggy 2024-08-09T14:52:37.600 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.375+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:52:37.600 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.375+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:52:37.600 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.375+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:52:37.600 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.375+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=59}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:52:37.600 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.375+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004650 con 0x55a86d4e4550 2024-08-09T14:52:37.600 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 16 ==== osd_map(59..63 src has 1..63) ==== 3511+0+0 (secure 0 0 0) 0x7fc3bc07e060 con 0x55a86d4642e0 2024-08-09T14:52:37.600 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(59..63 src has 1..63) 2024-08-09T14:52:37.600 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [59,63] > 58 2024-08-09T14:52:37.600 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 59 2024-08-09T14:52:37.600 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:37.600 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 59 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:37.600 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:37.601 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 60 2024-08-09T14:52:37.601 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:37.601 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 60 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:37.601 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:37.601 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 61 2024-08-09T14:52:37.650 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:52:37.645+0000 ea5d640 -1 osd.4 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.6s0 39.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-08-09T14:52:38.608 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:38.565+0000 12a65640 -1 osd.0 61 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:38.891 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:52:38.882+0000 ea5d640 -1 osd.4 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4491.0:1 39.6s0 39.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e32) 2024-08-09T14:52:39.768 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:39.753+0000 12a65640 -1 osd.0 62 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:40.056 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:39.940+0000 ea5d640 -1 osd.1 62 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:40.983 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:40.975+0000 ea5d640 -1 osd.1 63 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:41.063 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:40.941+0000 12a65640 -1 osd.0 63 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:42.101 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:42.093+0000 ea5d640 -1 osd.1 63 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:42.288 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:42.159+0000 12a65640 -1 osd.0 64 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4474.0:2 19.1d 19:bd63793a:test-rados-api-smithi155-41841-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-08-09T14:52:42.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:42.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 61 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:42.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:42.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 62 2024-08-09T14:52:42.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:42.377 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 62 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:42.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:42.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 63 2024-08-09T14:52:42.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:42.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 63 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:42.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:42.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:52:42.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:37.597+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:52:42.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:38.336+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:42.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:38.336+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:42.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:38.336+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:42.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:39.336+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:42.378 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:39.336+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:42.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:39.336+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:42.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:40.336+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:42.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:40.336+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:42.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:40.336+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:42.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:40.336+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:52:42.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:41.336+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:42.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:41.336+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:42.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:41.336+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:42.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.336+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:42.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.336+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:42.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.336+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:42.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.376+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:52:42.379 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.376+0000 7fc3cb426640 2 client.4435.objecter tid 2 on osd.0 is laggy 2024-08-09T14:52:42.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.376+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:52:42.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.376+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:52:42.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.376+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:52:43.509 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:43.370+0000 12a65640 -1 osd.0 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 20.1 20.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:43.650 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:43.485+0000 ea5d640 -1 osd.1 65 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:44.527 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:44.489+0000 12a65640 -1 osd.0 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 20.1 20.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:45.343 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:45.288+0000 ea5d640 -1 osd.1 66 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:45.545 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:45.514+0000 12a65640 -1 osd.0 65 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 20.1 20.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:46.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.376+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=64}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:52:46.338 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.376+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004820 con 0x55a86d4e4550 2024-08-09T14:52:46.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.379+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 17 ==== osd_map(64..66 src has 1..66) ==== 4470+0+0 (secure 0 0 0) 0x7fc3bc059440 con 0x55a86d4642e0 2024-08-09T14:52:46.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.379+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(64..66 src has 1..66) 2024-08-09T14:52:46.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.379+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [64,66] > 63 2024-08-09T14:52:46.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.379+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 64 2024-08-09T14:52:46.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.380+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:46.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.380+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 64 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:46.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.380+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:46.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.380+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 65 2024-08-09T14:52:46.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.380+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:46.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.380+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 65 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:46.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.380+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:46.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.380+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 66 2024-08-09T14:52:46.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.380+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:46.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.380+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 66 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:46.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.380+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:46.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.380+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:52:46.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:42.380+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:52:46.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:43.336+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:46.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:43.336+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:46.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:43.336+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:46.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:44.336+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:46.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:44.336+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:46.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:44.336+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:46.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:45.336+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:46.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:45.336+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:46.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:45.336+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:46.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:46.337+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:46.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:46.337+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:46.665 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:46.633+0000 12a65640 -1 osd.0 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 20.1 20.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:46.858 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:46.687+0000 ea5d640 -1 osd.1 68 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:47.490 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:46.337+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:47.490 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.337+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:47.490 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.337+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:47.490 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.337+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:47.490 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.377+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:52:47.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.377+0000 7fc3cb426640 2 client.4435.objecter tid 2 on osd.0 is laggy 2024-08-09T14:52:47.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.377+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:52:47.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.377+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:52:47.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.377+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:52:47.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.377+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=67}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:52:47.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.377+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004820 con 0x55a86d4e4550 2024-08-09T14:52:47.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.488+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 18 ==== osd_map(67..71 src has 1..71) ==== 3408+0+0 (secure 0 0 0) 0x7fc3bc07e070 con 0x55a86d4642e0 2024-08-09T14:52:47.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.488+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(67..71 src has 1..71) 2024-08-09T14:52:47.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.488+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [67,71] > 66 2024-08-09T14:52:47.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.488+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 67 2024-08-09T14:52:47.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.488+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:47.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 67 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:47.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:47.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 68 2024-08-09T14:52:47.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:47.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 68 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:47.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:47.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 69 2024-08-09T14:52:47.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:47.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 69 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:47.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:47.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 70 2024-08-09T14:52:47.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:47.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 70 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:47.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:47.734+0000 ea5d640 -1 osd.1 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:47.739 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:47.730+0000 12a65640 -1 osd.0 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 20.1 20.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:48.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 20 client. api_pool: Running main() from gmock_main.cc 2024-08-09T14:52:48.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-08-09T14:52:48.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [----------] Global test environment set-up. 2024-08-09T14:52:48.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-08-09T14:52:48.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-08-09T14:52:48.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ OK ] LibRadosPools.PoolList (5850 ms) 2024-08-09T14:52:48.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-08-09T14:52:48.608 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (5633 ms) 2024-08-09T14:52:48.608 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-08-09T14:52:48.608 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (6168 ms) 2024-08-09T14:52:48.608 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-08-09T14:52:48.608 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (4369 ms) 2024-08-09T14:52:48.608 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-08-09T14:52:48.609 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (4636 ms) 2024-08-09T14:52:48.609 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-08-09T14:52:48.609 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (11771 ms) 2024-08-09T14:52:48.609 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-08-09T14:52:48.609 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (6063 ms) 2024-08-09T14:52:48.609 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-08-09T14:52:48.609 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5874 ms) 2024-08-09T14:52:48.609 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-08-09T14:52:48.609 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (10974 ms) 2024-08-09T14:52:48.609 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [----------] 9 tests from LibRadosPools (61338 ms total) 2024-08-09T14:52:48.609 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: 2024-08-09T14:52:48.610 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [----------] Global test environment tear-down 2024-08-09T14:52:48.610 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (61339 ms total) 2024-08-09T14:52:48.610 INFO:tasks.workunit.client.0.smithi155.stdout: api_pool: [ PASSED ] 9 tests. 2024-08-09T14:52:48.869 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:48.788+0000 12a65640 -1 osd.0 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 20.1 20.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:48.988 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:48.976+0000 ea5d640 -1 osd.1 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:49.468 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-08-09T14:52:49.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [----------] Global test environment set-up. 2024-08-09T14:52:49.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-08-09T14:52:49.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-08-09T14:52:49.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-08-09T14:52:49.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-08-09T14:52:49.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: 2024-08-09T14:52:49.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-08-09T14:52:49.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-08-09T14:52:49.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-08-09T14:52:49.469 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: 2024-08-09T14:51:47.868+0000 7fccaba3f780 -1 failed for service _ceph-mon._tcp 2024-08-09T14:52:49.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: 2024-08-09T14:51:47.868+0000 7fccaba3f780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-08-09T14:52:49.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (618 ms) 2024-08-09T14:52:49.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-08-09T14:52:49.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5006 ms) 2024-08-09T14:52:49.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5624 ms total) 2024-08-09T14:52:49.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: 2024-08-09T14:52:49.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-08-09T14:52:49.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-08-09T14:52:49.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2bee490 2024-08-09T14:52:49.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2aa2980 2024-08-09T14:52:49.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e29067b0 2024-08-09T14:52:49.470 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e290a850 2024-08-09T14:52:49.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e3538670 2024-08-09T14:52:49.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c99410 2024-08-09T14:52:49.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c99e80 2024-08-09T14:52:49.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9a980 2024-08-09T14:52:49.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9b4f0 2024-08-09T14:52:49.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9c030 2024-08-09T14:52:49.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9cb20 2024-08-09T14:52:49.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9bd40 2024-08-09T14:52:49.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9da40 2024-08-09T14:52:49.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9c880 2024-08-09T14:52:49.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9e910 2024-08-09T14:52:49.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9d780 2024-08-09T14:52:49.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9f4c0 2024-08-09T14:52:49.471 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2a653d0 2024-08-09T14:52:49.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2a65e10 2024-08-09T14:52:49.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9e6f0 2024-08-09T14:52:49.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c979e0 2024-08-09T14:52:49.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2bee080 2024-08-09T14:52:49.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9ba20 2024-08-09T14:52:49.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c97af0 2024-08-09T14:52:49.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c97c20 2024-08-09T14:52:49.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c99f90 2024-08-09T14:52:49.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9af40 2024-08-09T14:52:49.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9a0e0 2024-08-09T14:52:49.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2b568c0 2024-08-09T14:52:49.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9bb30 2024-08-09T14:52:49.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c97d30 2024-08-09T14:52:49.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9a1f0 2024-08-09T14:52:49.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9a300 2024-08-09T14:52:49.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c99520 2024-08-09T14:52:49.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9b620 2024-08-09T14:52:49.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9c5f0 2024-08-09T14:52:49.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9b7d0 2024-08-09T14:52:49.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9b8e0 2024-08-09T14:52:49.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9aa90 2024-08-09T14:52:49.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9abe0 2024-08-09T14:52:49.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9acf0 2024-08-09T14:52:49.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9ae00 2024-08-09T14:52:49.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c98290 2024-08-09T14:52:49.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c983e0 2024-08-09T14:52:49.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c98590 2024-08-09T14:52:49.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9c140 2024-08-09T14:52:49.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c996d0 2024-08-09T14:52:49.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c997e0 2024-08-09T14:52:49.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c99930 2024-08-09T14:52:49.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2a65990 2024-08-09T14:52:49.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2a65b60 2024-08-09T14:52:49.804 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:49.797+0000 12a65640 -1 osd.0 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 20.1 20.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:50.143 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:50.058+0000 ea5d640 -1 osd.1 71 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:50.897 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:50.877+0000 12a65640 -1 osd.0 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 20.1 20.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:51.169 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:51.154+0000 ea5d640 -1 osd.1 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:51.925 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:51.917+0000 12a65640 -1 osd.0 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 20.1 20.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:52.167 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:27.410 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:52.179 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:27.423 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:52.194 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:27.441 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:52.204 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:27.448 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:52.218 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:27.465 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:52.258 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:27.501 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:52.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: starte4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:52.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 71 2024-08-09T14:52:52.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:52.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 71 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:52.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:52.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:52:52.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:47.489+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:52:52.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:48.337+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:52.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:48.337+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:52.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:48.337+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:52.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:49.337+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:52.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:49.337+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:52.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:49.337+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:52.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:50.337+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:52.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:50.337+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:52.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:50.337+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:52.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:50.337+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:52:52.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:51.337+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:52.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:51.337+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:52.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:51.337+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:52.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.337+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:52.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.337+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:52.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.337+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:52.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.378+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:52:52.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.378+0000 7fc3cb426640 2 client.4435.objecter tid 2 on osd.0 is laggy 2024-08-09T14:52:52.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.378+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:52:52.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.378+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:52:52.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.378+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:52:52.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.378+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=72}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:52:52.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.378+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004820 con 0x55a86d4e4550 2024-08-09T14:52:52.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 19 ==== osd_map(72..74 src has 1..74) ==== 816+0+0 (secure 0 0 0) 0x7fc3bc082070 con 0x55a86d4642e0 2024-08-09T14:52:52.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(72..74 src has 1..74) 2024-08-09T14:52:52.406 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:27.653 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:52.430 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:27.674 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:52.483 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:27.726 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:52:52.519 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:52.508+0000 ea5d640 -1 osd.1 73 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24.aabc5e21 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:53.075 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:53.006+0000 12a65640 -1 osd.0 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4435.0:2 20.1 20.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:53.739 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:53.695+0000 ea5d640 -1 osd.1 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:54.244 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:54.194+0000 12a65640 -1 osd.0 72 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4435.0:2 20.1 20.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:55.590 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:55.585+0000 ea5d640 -1 osd.1 75 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:55.840 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:55.775+0000 12a65640 -1 osd.0 73 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4435.0:2 20.1 20.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:56.906 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:56.881+0000 12a65640 -1 osd.0 74 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4435.0:2 20.1 20.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:57.003 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:56.943+0000 ea5d640 -1 osd.1 76 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:57.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [72,74] > 71 2024-08-09T14:52:57.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 72 2024-08-09T14:52:57.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:57.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 72 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:57.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:57.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 73 2024-08-09T14:52:57.380 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:57.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 73 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:57.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:57.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 74 2024-08-09T14:52:57.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:57.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 74 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:57.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:57.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:52:57.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:52.394+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:52:57.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:53.337+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:57.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:53.337+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:57.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:53.337+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:57.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:54.337+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:57.381 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:54.337+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:57.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:54.337+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:57.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:55.337+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:57.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:55.337+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:57.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:55.337+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:57.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:56.337+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:57.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:56.337+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:57.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:56.337+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:57.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.338+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:57.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.338+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:57.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.338+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:57.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.379+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:52:57.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.379+0000 7fc3cb426640 2 client.4435.objecter tid 2 on osd.0 is laggy 2024-08-09T14:52:57.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.379+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:52:57.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.379+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:52:57.985 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:57.976+0000 ea5d640 -1 osd.1 76 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:58.211 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:58.151+0000 12a65640 -1 osd.0 74 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4435.0:2 20.1 20.d3a59201 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:59.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.379+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:52:59.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.379+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=75}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:52:59.339 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.379+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004820 con 0x55a86d4e4550 2024-08-09T14:52:59.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 20 ==== osd_map(75..78 src has 1..78) ==== 1430+0+0 (secure 0 0 0) 0x7fc3bc082030 con 0x55a86d4642e0 2024-08-09T14:52:59.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(75..78 src has 1..78) 2024-08-09T14:52:59.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [75,78] > 74 2024-08-09T14:52:59.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 75 2024-08-09T14:52:59.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:59.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 75 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:59.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:59.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 76 2024-08-09T14:52:59.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:59.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 76 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:59.340 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:59.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 77 2024-08-09T14:52:59.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:59.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 77 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:59.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:59.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 78 2024-08-09T14:52:59.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:52:59.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 78 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:52:59.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:52:59.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:52:59.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:57.382+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:52:59.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:58.338+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:59.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:58.338+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:59.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:58.338+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:52:59.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:59.338+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:52:59.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:59.338+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:52:59.516 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:52:59.441+0000 12a65640 -1 osd.0 75 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4435.0:2 20.1 20:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:52:59.715 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:52:59.707+0000 ea5d640 -1 osd.1 78 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:53:00.645 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:53:00.570+0000 12a65640 -1 osd.0 75 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4435.0:2 20.1 20:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:53:00.872 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:53:00.860+0000 ea5d640 -1 osd.3 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4744.0:2 71.0 71:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57) 2024-08-09T14:53:01.266 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:53:01.174+0000 ea5d640 -1 osd.1 80 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4516.0:2 24.1 24:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:53:01.843 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:53:01.773+0000 12a65640 -1 osd.0 76 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4435.0:2 20.1 20:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:53:02.122 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:53:02.008+0000 ea5d640 -1 osd.3 79 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4744.0:2 71.0 71:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57) 2024-08-09T14:53:02.193 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:53:02.185+0000 ea5d640 -1 osd.1 80 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4516.0:2 24.1 24:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:53:02.385 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:52:59.338+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:02.385 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:00.338+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:02.385 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:00.338+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:02.385 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:00.338+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:02.385 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:00.338+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:53:02.385 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:00.649+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 2 ==== osd_map(36..74 src has 1..74) ==== 33345+0+0 (crc 0 0 0) 0x7fc3c00084b0 con 0x55a86d4e4550 2024-08-09T14:53:02.385 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:00.649+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(36..74 src has 1..74) 2024-08-09T14:53:02.385 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:00.649+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map ignoring epochs [36,74] <= 78 2024-08-09T14:53:02.385 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:00.649+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:53:02.385 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:00.649+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:53:02.385 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:01.338+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:02.386 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:01.338+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:02.386 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:01.338+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:02.386 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.338+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:02.386 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.338+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:02.386 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.338+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:02.386 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.380+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:53:02.386 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.380+0000 7fc3cb426640 2 client.4435.objecter tid 2 on osd.0 is laggy 2024-08-09T14:53:02.386 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.380+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:53:02.386 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.380+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:53:02.386 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.380+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:53:02.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.380+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=79}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:53:02.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.380+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004820 con 0x55a86d4e4550 2024-08-09T14:53:02.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 21 ==== osd_map(79..82 src has 1..82) ==== 2537+0+0 (secure 0 0 0) 0x7fc3bc082070 con 0x55a86d4642e0 2024-08-09T14:53:02.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(79..82 src has 1..82) 2024-08-09T14:53:02.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [79,82] > 78 2024-08-09T14:53:02.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 79 2024-08-09T14:53:02.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:53:02.387 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 79 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:02.388 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:03.064 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:53:03.001+0000 12a65640 -1 osd.0 77 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4435.0:2 20.1 20:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:53:03.286 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:53:03.252+0000 ea5d640 -1 osd.3 80 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4744.0:2 71.0 71:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57) 2024-08-09T14:53:03.847 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:53:03.839+0000 ea5d640 -1 osd.1 82 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4516.0:2 24.1 24:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:53:04.151 INFO:tasks.workunit.client.0.smithi155.stdout: d 0x5606e2a65c70 2024-08-09T14:53:04.151 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2bed830 2024-08-09T14:53:04.151 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9c250 2024-08-09T14:53:04.151 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9f380 2024-08-09T14:53:04.151 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9c3a0 2024-08-09T14:53:04.151 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9d590 2024-08-09T14:53:04.152 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c98b20 2024-08-09T14:53:04.152 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2bed940 2024-08-09T14:53:04.152 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2beda50 2024-08-09T14:53:04.152 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2bedb60 2024-08-09T14:53:04.152 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c9e4b0 2024-08-09T14:53:04.152 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c95980 2024-08-09T14:53:04.152 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 0x5606e2c95f80 2024-08-09T14:53:04.152 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: started 64 aios 2024-08-09T14:53:04.152 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2bee490 2024-08-09T14:53:04.152 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2aa2980 2024-08-09T14:53:04.152 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e29067b0 2024-08-09T14:53:04.152 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e290a850 2024-08-09T14:53:04.152 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e3538670 2024-08-09T14:53:04.153 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c99410 2024-08-09T14:53:04.153 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c99e80 2024-08-09T14:53:04.153 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9a980 2024-08-09T14:53:04.153 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9b4f0 2024-08-09T14:53:04.153 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9c030 2024-08-09T14:53:04.153 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9cb20 2024-08-09T14:53:04.153 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9bd40 2024-08-09T14:53:04.153 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9da40 2024-08-09T14:53:04.153 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9c880 2024-08-09T14:53:04.153 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9e910 2024-08-09T14:53:04.153 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9d780 2024-08-09T14:53:04.154 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9f4c0 2024-08-09T14:53:04.154 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2a653d0 2024-08-09T14:53:04.154 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2a65e10 2024-08-09T14:53:04.154 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9e6f0 2024-08-09T14:53:04.154 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c979e0 2024-08-09T14:53:04.154 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2bee080 2024-08-09T14:53:04.154 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9ba20 2024-08-09T14:53:04.154 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c97af0 2024-08-09T14:53:04.154 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c97c20 2024-08-09T14:53:04.154 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c99f90 2024-08-09T14:53:04.154 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9af40 2024-08-09T14:53:04.154 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9a0e0 2024-08-09T14:53:04.154 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2b568c0 2024-08-09T14:53:04.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9bb30 2024-08-09T14:53:04.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c97d30 2024-08-09T14:53:04.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9a1f0 2024-08-09T14:53:04.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9a300 2024-08-09T14:53:04.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c99520 2024-08-09T14:53:04.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9b620 2024-08-09T14:53:04.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9c5f0 2024-08-09T14:53:04.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9b7d0 2024-08-09T14:53:04.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9b8e0 2024-08-09T14:53:04.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9aa90 2024-08-09T14:53:04.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9abe0 2024-08-09T14:53:04.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9acf0 2024-08-09T14:53:04.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9ae00 2024-08-09T14:53:04.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c98290 2024-08-09T14:53:04.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c983e0 2024-08-09T14:53:04.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c98590 2024-08-09T14:53:04.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9c140 2024-08-09T14:53:04.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c996d0 2024-08-09T14:53:04.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c997e0 2024-08-09T14:53:04.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c99930 2024-08-09T14:53:04.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2a65990 2024-08-09T14:53:04.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2a65b60 2024-08-09T14:53:04.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2a65c70 2024-08-09T14:53:04.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2bed830 2024-08-09T14:53:04.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9c250 2024-08-09T14:53:04.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9f380 2024-08-09T14:53:04.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9c3a0 2024-08-09T14:53:04.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9d590 2024-08-09T14:53:04.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c98b20 2024-08-09T14:53:04.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2bed940 2024-08-09T14:53:04.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2beda50 2024-08-09T14:53:04.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2bedb60 2024-08-09T14:53:04.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c9e4b0 2024-08-09T14:53:04.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c95980 2024-08-09T14:53:04.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: waiting 0x5606e2c95f80 2024-08-09T14:53:04.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: done. 2024-08-09T14:53:04.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (71276 ms) 2024-08-09T14:53:04.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (71276 ms total) 2024-08-09T14:53:04.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: 2024-08-09T14:53:04.286 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:53:04.189+0000 12a65640 -1 osd.0 78 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4435.0:2 20.1 20:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:53:05.354 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:53:05.121+0000 ea5d640 -1 osd.3 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4744.0:2 71.0 71:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57) 2024-08-09T14:53:06.144 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:53:06.018+0000 ea5d640 -1 osd.1 84 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4516.0:2 24.1 24:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:53:06.439 INFO:tasks.workunit.client.0.smithi155.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-08-09T14:53:06.440 INFO:tasks.workunit.client.0.smithi155.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-08-09T14:53:06.440 INFO:tasks.workunit.client.0.smithi155.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-08-09T14:53:06.440 INFO:tasks.workunit.client.0.smithi155.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-08-09T14:53:06.440 INFO:tasks.workunit.client.0.smithi155.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (78916 ms) 2024-08-09T14:53:06.440 INFO:tasks.workunit.client.0.smithi155.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (78916 ms total) 2024-08-09T14:53:06.440 INFO:tasks.workunit.client.0.smithi155.stdout: api_cls_remote_reads: 2024-08-09T14:53:06.440 INFO:tasks.workunit.client.0.smithi155.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-08-09T14:53:06.440 INFO:tasks.workunit.client.0.smithi155.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (78916 ms total) 2024-08-09T14:53:06.440 INFO:tasks.workunit.client.0.smithi155.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-08-09T14:53:07.105 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:53:07.071+0000 ea5d640 -1 osd.1 84 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4516.0:2 24.1 24:847a3d55:::obj:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e30) 2024-08-09T14:53:07.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 80 2024-08-09T14:53:07.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:53:07.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 80 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:07.382 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:07.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 81 2024-08-09T14:53:07.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:53:07.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 81 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:07.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:07.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 82 2024-08-09T14:53:07.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:53:07.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 82 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:07.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:07.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:53:07.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:02.383+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:53:07.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:03.338+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:07.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:03.338+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:07.383 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:03.338+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:07.384 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:04.338+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:07.384 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:04.338+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:07.384 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:04.338+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:07.384 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:05.338+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:07.384 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:05.338+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:07.384 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:05.338+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:07.384 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:06.338+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:07.384 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:06.338+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:07.384 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:06.338+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:07.384 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.338+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:07.384 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.338+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:07.384 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.338+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:07.384 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.381+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:53:07.385 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.381+0000 7fc3cb426640 2 client.4435.objecter tid 2 on osd.0 is laggy 2024-08-09T14:53:07.385 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.381+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:53:07.385 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.381+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:53:08.026 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.381+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:53:08.026 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.381+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=83}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:53:08.026 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.381+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004820 con 0x55a86d4e4550 2024-08-09T14:53:08.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 22 ==== osd_map(83..87 src has 1..87) ==== 6128+0+0 (secure 0 0 0) 0x7fc3bc082070 con 0x55a86d4642e0 2024-08-09T14:53:08.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(83..87 src has 1..87) 2024-08-09T14:53:08.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [83,87] > 82 2024-08-09T14:53:08.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 83 2024-08-09T14:53:08.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:53:08.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 83 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:08.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:08.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 84 2024-08-09T14:53:08.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:53:08.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 84 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:08.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:08.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 85 2024-08-09T14:53:08.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:53:08.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 85 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:08.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:08.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 86 2024-08-09T14:53:08.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:53:08.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 86 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:08.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:08.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 87 2024-08-09T14:53:08.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 10 client.4435.objecter checking op 2 2024-08-09T14:53:08.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 87 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:08.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:08.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:53:08.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:07.422+0000 7fc3ba7fc640 20 client.4435.objecter 2 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:53:08.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 3 ==== osd_op_reply(2 exist [writefull 0~5] v74'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:53:08.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(2 exist [writefull 0~5] v74'1 uv1 ondisk = 0) 2024-08-09T14:53:08.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:53:08.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 2 ondisk uv 1 in 20.1 attempt 0 2024-08-09T14:53:08.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 0 2024-08-09T14:53:08.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 2 2024-08-09T14:53:08.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 2 2024-08-09T14:53:08.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:53:08.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 2 2024-08-09T14:53:08.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3c99c3640 5 client.4435.objecter 0 in flight 2024-08-09T14:53:08.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 10 librados: Objecter returned from writefull r=0 2024-08-09T14:53:08.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-08-09T14:53:08.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d4e3b90 2024-08-09T14:53:08.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 87 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:53:08.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:08.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:53:08.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:53:08.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid exist '@20' '@20' [read 0~256] tid 3 osd.0 2024-08-09T14:53:08.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:53:08.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 3 2024-08-09T14:53:08.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 3 to 20.1 on osd.0 2024-08-09T14:53:08.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:3 20.1 20:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87) -- 0x55a86d4f1360 con 0x55a86d4e4550 2024-08-09T14:53:08.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:53:08.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 5 client.4435.objecter 1 in flight 2024-08-09T14:53:08.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d4f1700 2024-08-09T14:53:08.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 87 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:53:08.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2024-08-09T14:53:08.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.a7bbaa1 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:53:08.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:53:08.505 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T14:53:08.457+0000 ea5d640 -1 osd.1 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4619.0:2 62.6 62:602f83fe:::foo:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48) 2024-08-09T14:53:13.880 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid noexist '@20' '@20' [read 0~256] tid 4 osd.0 2024-08-09T14:53:13.880 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:53:13.880 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 4 2024-08-09T14:53:13.880 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 4 to 20.1 on osd.0 2024-08-09T14:53:13.880 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:4 20.1 20:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e87) -- 0x55a86d4f1bb0 con 0x55a86d4e4550 2024-08-09T14:53:13.880 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 5 2024-08-09T14:53:13.880 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.025+0000 7fc3cb6c6780 5 client.4435.objecter 2 in flight 2024-08-09T14:53:13.880 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.339+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:13.881 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.339+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:13.881 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:08.339+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:13.881 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:09.339+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:13.881 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:09.339+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:13.881 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:09.339+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:13.881 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:10.339+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:13.881 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:10.339+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:13.881 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:10.339+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:13.881 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:10.339+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:53:13.881 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:11.339+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:13.881 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:11.339+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:13.882 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:11.339+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:13.882 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:12.339+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:13.882 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:12.339+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:13.882 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:12.339+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:13.882 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:12.382+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:53:13.882 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.339+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:13.882 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.339+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:13.882 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.339+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:13.882 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 4 ==== osd_map(88..88 src has 1..89) ==== 631+0+0 (crc 0 0 0) 0x7fc3c00085b0 con 0x55a86d4e4550 2024-08-09T14:53:13.883 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 5 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:53:13.883 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(88..88 src has 1..89) 2024-08-09T14:53:13.883 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-08-09T14:53:13.883 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:53:13.883 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [88,88] > 87 2024-08-09T14:53:13.883 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 88 2024-08-09T14:53:13.883 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3ba7fc640 10 client.4435.objecter checking op 3 2024-08-09T14:53:13.883 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 88 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:53:13.883 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:13.883 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3ba7fc640 10 client.4435.objecter checking op 4 2024-08-09T14:53:13.884 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 88 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:53:13.884 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2024-08-09T14:53:13.884 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:53:13.884 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3ba7fc640 20 client.4435.objecter 3 20.d3a59201 osd.0 exist [read 0~256] 2024-08-09T14:53:13.884 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3ba7fc640 20 client.4435.objecter 4 20.a7bbaa1 osd.0 noexist [read 0~256] 2024-08-09T14:53:13.884 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 3 ondisk uv 1 in 20.1 attempt 0 2024-08-09T14:53:13.884 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 5 2024-08-09T14:53:13.884 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 3 2024-08-09T14:53:13.884 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 3 2024-08-09T14:53:13.884 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 5 2024-08-09T14:53:13.884 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 3 2024-08-09T14:53:13.885 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 5 client.4435.objecter 1 in flight 2024-08-09T14:53:13.885 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 6 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:53:13.885 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-09T14:53:13.885 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:53:13.885 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (5853 ms) 2024-08-09T14:53:13.885 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 4 ondisk uv 0 in 20.1 attempt 0 2024-08-09T14:53:13.885 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 10 client.4435.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-09T14:53:13.885 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-08-09T14:53:13.885 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 0 2024-08-09T14:53:13.886 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 4 2024-08-09T14:53:13.886 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 4 2024-08-09T14:53:13.886 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:53:13.886 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 4 2024-08-09T14:53:14.034 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3c99c3640 5 client.4435.objecter 0 in flight 2024-08-09T14:53:14.034 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d4f1700 2024-08-09T14:53:14.034 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 88 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:53:14.034 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:14.034 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:53:14.034 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:53:14.034 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid exist '@20' '@20' [read 0~256] tid 5 osd.0 2024-08-09T14:53:14.034 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:53:14.035 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 5 2024-08-09T14:53:14.035 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 5 to 20.1 on osd.0 2024-08-09T14:53:14.035 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:5 20.1 20:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88) -- 0x55a86d4f1f50 con 0x55a86d4e4550 2024-08-09T14:53:14.035 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:53:14.035 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 5 client.4435.objecter 1 in flight 2024-08-09T14:53:14.035 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d4e3b90 2024-08-09T14:53:14.035 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 88 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:53:14.035 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2024-08-09T14:53:14.035 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.a7bbaa1 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:53:14.035 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:53:14.035 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid noexist '@20' '@20' [read 0~256] tid 6 osd.0 2024-08-09T14:53:14.036 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:53:14.036 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 6 2024-08-09T14:53:14.036 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 6 to 20.1 on osd.0 2024-08-09T14:53:14.036 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:6 20.1 20:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88) -- 0x55a86d5a8f40 con 0x55a86d4e4550 2024-08-09T14:53:14.036 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 5 2024-08-09T14:53:14.036 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:13.878+0000 7fc3cb6c6780 5 client.4435.objecter 2 in flight 2024-08-09T14:53:14.036 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 7 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:53:14.036 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-08-09T14:53:14.036 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:53:14.036 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 5 ondisk uv 1 in 20.1 attempt 0 2024-08-09T14:53:14.036 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 5 2024-08-09T14:53:14.036 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 5 2024-08-09T14:53:14.037 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 5 2024-08-09T14:53:14.037 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 5 2024-08-09T14:53:14.037 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 5 2024-08-09T14:53:14.037 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 5 client.4435.objecter 1 in flight 2024-08-09T14:53:14.037 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 8 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:53:14.037 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-09T14:53:14.037 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:53:14.037 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 6 ondisk uv 0 in 20.1 attempt 0 2024-08-09T14:53:14.037 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 10 client.4435.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-09T14:53:14.037 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 0 2024-08-09T14:53:14.037 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 6 2024-08-09T14:53:14.037 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 6 2024-08-09T14:53:14.037 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:53:14.038 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 6 2024-08-09T14:53:14.038 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3c99c3640 5 client.4435.objecter 0 in flight 2024-08-09T14:53:14.038 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (154 ms) 2024-08-09T14:53:14.038 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-08-09T14:53:14.038 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d4e3b90 2024-08-09T14:53:14.038 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 88 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:53:14.038 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:14.038 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:53:14.038 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:53:14.038 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid exist '@20' '@20' [read 0~256] tid 7 osd.0 2024-08-09T14:53:14.038 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:53:14.038 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 7 2024-08-09T14:53:14.038 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 7 to 20.1 on osd.0 2024-08-09T14:53:16.066 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:7 20.1 20:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88) -- 0x55a86d4f1700 con 0x55a86d4e4550 2024-08-09T14:53:16.066 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:53:16.066 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 5 client.4435.objecter 1 in flight 2024-08-09T14:53:16.066 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d5e9970 2024-08-09T14:53:16.066 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 88 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:53:16.066 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2024-08-09T14:53:16.066 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.a7bbaa1 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:53:16.066 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:53:16.066 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid noexist '@20' '@20' [read 0~256] tid 8 osd.0 2024-08-09T14:53:16.066 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:53:16.067 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 8 2024-08-09T14:53:16.067 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 8 to 20.1 on osd.0 2024-08-09T14:53:16.067 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:8 20.1 20:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e88) -- 0x55a86d5e9e20 con 0x55a86d4e4550 2024-08-09T14:53:16.067 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 5 2024-08-09T14:53:16.067 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.032+0000 7fc3cb6c6780 5 client.4435.objecter 2 in flight 2024-08-09T14:53:16.067 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.339+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:16.067 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.339+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:16.067 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:14.339+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:16.067 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:15.339+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:16.067 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:15.339+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:16.067 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:15.339+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:16.067 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (2032 ms) 2024-08-09T14:53:16.068 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-08-09T14:53:16.068 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 9 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:53:16.068 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-08-09T14:53:16.068 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:53:16.068 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 7 ondisk uv 1 in 20.1 attempt 0 2024-08-09T14:53:16.068 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 5 2024-08-09T14:53:16.068 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 7 2024-08-09T14:53:16.068 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 7 2024-08-09T14:53:16.068 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 5 2024-08-09T14:53:16.068 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 7 2024-08-09T14:53:16.068 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 5 client.4435.objecter 1 in flight 2024-08-09T14:53:16.068 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 10 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:53:16.069 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-09T14:53:16.069 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:53:16.069 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 8 ondisk uv 0 in 20.1 attempt 0 2024-08-09T14:53:16.069 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 10 client.4435.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-09T14:53:16.069 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 0 2024-08-09T14:53:16.069 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 8 2024-08-09T14:53:16.069 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 8 2024-08-09T14:53:16.069 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:53:16.069 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 8 2024-08-09T14:53:16.069 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3c99c3640 5 client.4435.objecter 0 in flight 2024-08-09T14:53:16.069 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-08-09T14:53:16.069 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 20 librados: queue_aio_write 0x55a86d4f5580 completion 0x55a86d376850 write_seq 1 2024-08-09T14:53:16.069 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d4e3b90 2024-08-09T14:53:16.070 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 88 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:16.070 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:16.070 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:53:16.070 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:53:16.070 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid exist '@20' '@20' [write 0~5 in=5b] tid 9 osd.0 2024-08-09T14:53:16.070 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:53:16.070 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 9 2024-08-09T14:53:16.070 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 9 to 20.1 on osd.0 2024-08-09T14:53:16.070 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:9 20.1 20:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e88) -- 0x55a86d5aa2f0 con 0x55a86d4e4550 2024-08-09T14:53:17.427 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:52.670 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:17.454 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:52.698 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:17.509 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:52.752 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:17.512 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:52.759 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:17.513 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:52.756 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:17.528 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:52.775 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:17.680 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:03:52.927 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:17.707 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:03:52.950 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:17.824 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:03:53.067 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:20.032 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:53:20.012+0000 ea5d640 -1 osd.3 96 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4744.0:2 71.0 71:0384ab8e:::oid:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57) 2024-08-09T14:53:26.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:53:26.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 5 client.4435.objecter 1 in flight 2024-08-09T14:53:26.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.064+0000 7fc3cb6c6780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-08-09T14:53:26.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.339+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:26.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.339+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:26.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:16.339+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:26.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:17.339+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:26.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:17.339+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:26.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:17.339+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:26.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:17.383+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:53:26.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:18.339+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:26.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:18.339+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:26.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:18.339+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:26.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:19.340+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:26.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:19.340+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:26.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:19.340+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:26.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:20.340+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:26.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:20.340+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:26.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:20.340+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:26.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:20.340+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:53:26.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:21.340+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:26.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:21.340+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:26.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:21.340+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:26.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:22.340+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:26.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:22.340+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:26.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:22.340+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:26.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:22.384+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:53:26.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:23.341+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:26.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:23.341+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:26.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:23.341+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:26.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:24.340+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:26.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:24.340+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:26.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:24.340+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:26.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:25.340+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:26.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:25.340+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:26.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:25.340+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:26.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:26.340+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:26.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:26.340+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:27.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:26.340+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:27.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.340+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:27.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.340+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:27.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.340+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:27.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.384+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:53:27.491 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.384+0000 7fc3cb426640 2 client.4435.objecter tid 9 on osd.0 is laggy 2024-08-09T14:53:27.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.384+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:53:27.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.384+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:53:27.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.384+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:53:27.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.384+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=89}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:53:27.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.384+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c0047d0 con 0x55a86d4e4550 2024-08-09T14:53:27.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 23 ==== osd_map(89..105 src has 1..105) ==== 11406+0+0 (secure 0 0 0) 0x7fc3bc082070 con 0x55a86d4642e0 2024-08-09T14:53:27.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(89..105 src has 1..105) 2024-08-09T14:53:27.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [89,105] > 88 2024-08-09T14:53:27.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 89 2024-08-09T14:53:27.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:27.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 89 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:27.492 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:27.493 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 90 2024-08-09T14:53:27.493 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:27.493 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 90 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:27.493 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:27.493 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 91 2024-08-09T14:53:27.493 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:27.493 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 91 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:27.493 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:27.493 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 92 2024-08-09T14:53:27.493 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:27.493 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 92 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:27.493 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:27.493 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 93 2024-08-09T14:53:27.494 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:27.494 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 93 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:27.494 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:27.494 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 94 2024-08-09T14:53:27.494 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:27.494 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 94 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:27.494 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:27.494 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 95 2024-08-09T14:53:27.494 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:27.494 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 95 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:27.494 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:27.494 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 96 2024-08-09T14:53:27.495 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:27.495 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 96 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:27.495 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:27.495 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 97 2024-08-09T14:53:27.495 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:27.495 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 97 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:27.495 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:27.495 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 98 2024-08-09T14:53:27.495 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:27.495 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 98 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:27.495 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:27.495 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 99 2024-08-09T14:53:27.496 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:27.496 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 99 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:27.496 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:27.496 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 100 2024-08-09T14:53:30.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:30.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 100 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:30.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:30.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 101 2024-08-09T14:53:30.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:30.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 101 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:30.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:30.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 102 2024-08-09T14:53:30.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:30.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 102 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:30.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:30.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.489+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 103 2024-08-09T14:53:30.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.490+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:30.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.490+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 103 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:30.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.490+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:30.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.490+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 104 2024-08-09T14:53:30.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.490+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:30.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.490+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 104 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:30.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.490+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:30.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.490+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 105 2024-08-09T14:53:30.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.490+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:30.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.490+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 105 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:30.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.490+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:30.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.490+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:53:30.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:27.490+0000 7fc3ba7fc640 20 client.4435.objecter 9 20.d3a59201 osd.0 exist [write 0~5 in=5b] 2024-08-09T14:53:30.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:28.340+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:30.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:28.340+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:30.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:28.340+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:30.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:29.340+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:30.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:29.340+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:30.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:29.340+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:32.390 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:30.341+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:32.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:30.341+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:32.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:30.341+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:32.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:30.341+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:53:32.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:31.341+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:32.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:31.341+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:32.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:31.341+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:32.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.341+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:32.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.341+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:32.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.341+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:32.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.385+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:53:32.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.385+0000 7fc3cb426640 2 client.4435.objecter tid 9 on osd.0 is laggy 2024-08-09T14:53:32.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.385+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:53:32.391 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.385+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:53:32.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.385+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:53:32.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.385+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=106}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:53:32.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.385+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c0047d0 con 0x55a86d4e4550 2024-08-09T14:53:32.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 24 ==== osd_map(106..109 src has 1..109) ==== 1774+0+0 (secure 0 0 0) 0x7fc3bc082050 con 0x55a86d4642e0 2024-08-09T14:53:32.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(106..109 src has 1..109) 2024-08-09T14:53:32.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [106,109] > 105 2024-08-09T14:53:32.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 106 2024-08-09T14:53:32.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:32.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 106 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:32.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:32.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 107 2024-08-09T14:53:32.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:32.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 107 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:32.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:32.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 108 2024-08-09T14:53:32.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:37.503 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 108 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:37.503 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:37.503 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 109 2024-08-09T14:53:37.503 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:37.503 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 109 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:37.503 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:37.503 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:53:37.504 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:32.389+0000 7fc3ba7fc640 20 client.4435.objecter 9 20.d3a59201 osd.0 exist [write 0~5 in=5b] 2024-08-09T14:53:37.504 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:33.341+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:37.504 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:33.341+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:37.504 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:33.341+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:37.504 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:34.341+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:37.504 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:34.341+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:37.504 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:34.341+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:37.504 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:35.341+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:37.504 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:35.341+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:37.504 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:35.341+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:37.504 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:36.341+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:37.504 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:36.341+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:37.504 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:36.341+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:37.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.341+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:37.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.341+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:37.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.341+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:37.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.386+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:53:37.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.386+0000 7fc3cb426640 2 client.4435.objecter tid 9 on osd.0 is laggy 2024-08-09T14:53:37.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.386+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:53:37.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.386+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:53:37.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.386+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:53:37.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.386+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=110}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:53:37.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.386+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c0047d0 con 0x55a86d4e4550 2024-08-09T14:53:37.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 25 ==== osd_map(110..114 src has 1..114) ==== 6293+0+0 (secure 0 0 0) 0x7fc3bc082030 con 0x55a86d4642e0 2024-08-09T14:53:37.505 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(110..114 src has 1..114) 2024-08-09T14:53:37.532 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 watch_notify: Running main() from gmock_main.cc 2024-08-09T14:53:37.532 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-08-09T14:53:37.532 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: [----------] Global test environment set-up. 2024-08-09T14:53:37.532 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-08-09T14:53:37.532 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-08-09T14:53:37.533 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: handle_notify cookie 93975606077712 notify_id 167503724544 notifier_gid 4466 2024-08-09T14:53:37.533 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (26174 ms) 2024-08-09T14:53:37.533 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-08-09T14:53:37.533 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: Trying... 2024-08-09T14:53:37.533 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: handle_notify cookie 93975619526368 notify_id 416611827713 notifier_gid 4619 2024-08-09T14:53:37.533 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: Waiting for 3.000000000s 2024-08-09T14:53:37.533 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: Timed out. 2024-08-09T14:53:37.533 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: Flushing... 2024-08-09T14:53:37.533 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: Flushed... 2024-08-09T14:53:37.533 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (83650 ms) 2024-08-09T14:53:37.533 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (109824 ms total) 2024-08-09T14:53:37.533 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: 2024-08-09T14:53:37.534 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: [----------] Global test environment tear-down 2024-08-09T14:53:37.534 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (109824 ms total) 2024-08-09T14:53:37.534 INFO:tasks.workunit.client.0.smithi155.stdout: watch_notify: [ PASSED ] 2 tests. 2024-08-09T14:53:40.343 INFO:tasks.workunit.client.0.smithi155.stdout:3 client.4435.objecter handle_osd_map got epochs [110,114] > 109 2024-08-09T14:53:40.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 110 2024-08-09T14:53:40.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:40.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 110 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:40.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:40.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 111 2024-08-09T14:53:40.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:40.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 111 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:40.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:40.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 112 2024-08-09T14:53:40.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:40.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 112 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:40.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:40.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 113 2024-08-09T14:53:40.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:40.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 113 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:40.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:40.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 114 2024-08-09T14:53:40.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:40.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 114 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:40.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:40.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:53:40.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:37.501+0000 7fc3ba7fc640 20 client.4435.objecter 9 20.d3a59201 osd.0 exist [write 0~5 in=5b] 2024-08-09T14:53:40.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:38.341+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:40.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:38.341+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:40.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:38.341+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:40.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:39.342+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:40.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:39.342+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:40.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:39.342+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:40.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:40.342+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:40.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:40.342+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:40.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:40.342+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:42.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:40.342+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:53:42.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:41.342+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:42.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:41.342+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:42.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:41.342+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:42.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.342+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:42.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.342+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:42.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.342+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:42.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.387+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:53:42.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.387+0000 7fc3cb426640 2 client.4435.objecter tid 9 on osd.0 is laggy 2024-08-09T14:53:42.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.387+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:53:42.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.387+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:53:42.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.387+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:53:42.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.387+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=115}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:53:42.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.387+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c0047d0 con 0x55a86d4e4550 2024-08-09T14:53:42.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 26 ==== osd_map(115..118 src has 1..118) ==== 3911+0+0 (secure 0 0 0) 0x7fc3bc082070 con 0x55a86d4642e0 2024-08-09T14:53:42.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(115..118 src has 1..118) 2024-08-09T14:53:42.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [115,118] > 114 2024-08-09T14:53:42.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 115 2024-08-09T14:53:42.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:42.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 115 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:42.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:42.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 116 2024-08-09T14:53:42.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:42.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 116 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:42.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:42.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 117 2024-08-09T14:53:42.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:42.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 117 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:42.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:42.470 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:53:42.464+0000 12a65640 -1 osd.0 114 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:7 25.d 25:b46e3eb9:::5.obj:head [write 0~155 in=155b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:53:43.692 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:53:43.653+0000 ea5d640 -1 osd.5 115 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:2 82.1e 82.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e89) 2024-08-09T14:53:43.893 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:53:43.797+0000 12a65640 -1 osd.0 116 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:7 25.d 25:b46e3eb9:::5.obj:head [write 0~155 in=155b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:53:44.818 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:53:44.796+0000 12a65640 -1 osd.0 116 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:7 25.d 25:b46e3eb9:::5.obj:head [write 0~155 in=155b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:53:45.065 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:53:45.051+0000 ea5d640 -1 osd.5 116 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:2 82.1e 82.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e89) 2024-08-09T14:53:45.976 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:53:45.968+0000 12a65640 -1 osd.0 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:7 25.d 25:b46e3eb9:::5.obj:head [write 0~155 in=155b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:53:46.282 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:53:46.166+0000 ea5d640 -1 osd.5 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:2 82.1e 82.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e89) 2024-08-09T14:53:47.204 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:53:47.135+0000 12a65640 -1 osd.0 119 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:7 25.d 25:b46e3eb9:::5.obj:head [write 0~155 in=155b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:53:47.298 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:53:47.221+0000 ea5d640 -1 osd.5 118 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:2 82.1e 82.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e89) 2024-08-09T14:53:47.535 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 118 2024-08-09T14:53:47.535 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:47.535 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 118 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:47.535 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:47.535 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:53:47.536 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:42.423+0000 7fc3ba7fc640 20 client.4435.objecter 9 20.d3a59201 osd.0 exist [write 0~5 in=5b] 2024-08-09T14:53:47.536 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:43.342+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:47.536 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:43.342+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:47.536 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:43.342+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:47.536 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:44.342+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:47.536 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:44.342+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:47.536 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:44.342+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:47.536 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:45.342+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:47.536 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:45.342+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:47.536 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:45.342+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:47.536 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:46.342+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:47.537 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:46.342+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:47.537 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:46.342+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:47.537 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.342+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:47.537 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.342+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:47.537 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.342+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:47.537 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.388+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:53:47.537 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.388+0000 7fc3cb426640 2 client.4435.objecter tid 9 on osd.0 is laggy 2024-08-09T14:53:47.537 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.388+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:53:47.537 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.388+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:53:47.537 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.388+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:53:47.537 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.388+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=119}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:53:47.537 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.388+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c0047d0 con 0x55a86d4e4550 2024-08-09T14:53:47.537 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 27 ==== osd_map(119..121 src has 1..121) ==== 840+0+0 (secure 0 0 0) 0x7fc3bc082020 con 0x55a86d4642e0 2024-08-09T14:53:47.537 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(119..121 src has 1..121) 2024-08-09T14:53:47.538 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [119,121] > 118 2024-08-09T14:53:47.538 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 119 2024-08-09T14:53:48.263 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:53:48.250+0000 12a65640 -1 osd.0 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:7 25.d 25:b46e3eb9:::5.obj:head [write 0~155 in=155b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:53:48.458 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:53:48.360+0000 ea5d640 -1 osd.5 120 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:2 82.1e 82.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e89) 2024-08-09T14:53:49.640 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:53:49.634+0000 12a65640 -1 osd.0 121 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:7 25.d 25:b46e3eb9:::5.obj:head [write 0~155 in=155b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:53:51.009 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:53:50.945+0000 12a65640 -1 osd.0 122 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:7 25.d 25:b46e3eb9:::5.obj:head [write 0~155 in=155b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:53:52.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:52.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 119 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:52.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:52.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 120 2024-08-09T14:53:52.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:52.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 120 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:52.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:52.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 121 2024-08-09T14:53:52.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:52.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 121 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:52.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:52.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:53:52.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:47.534+0000 7fc3ba7fc640 20 client.4435.objecter 9 20.d3a59201 osd.0 exist [write 0~5 in=5b] 2024-08-09T14:53:52.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:48.342+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:52.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:48.342+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:52.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:48.342+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:52.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:49.342+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:52.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:49.342+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:52.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:49.342+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:52.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:50.343+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:52.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:50.343+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:52.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:50.343+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:52.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:50.343+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:53:52.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:50.734+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 11 ==== osd_map(89..119 src has 1..121) ==== 23544+0+0 (crc 0 0 0) 0x7fc3c00085b0 con 0x55a86d4e4550 2024-08-09T14:53:52.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:50.734+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(89..119 src has 1..121) 2024-08-09T14:53:52.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:50.734+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map ignoring epochs [89,119] <= 121 2024-08-09T14:53:52.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:50.734+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:53:52.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:50.734+0000 7fc3ba7fc640 20 client.4435.objecter 9 20.d3a59201 osd.0 exist [write 0~5 in=5b] 2024-08-09T14:53:52.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:51.343+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:52.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:51.343+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:52.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:51.343+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:52.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.343+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:52.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.343+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:52.573 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.343+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:52.574 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.389+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:53:52.574 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.389+0000 7fc3cb426640 2 client.4435.objecter tid 9 on osd.0 is laggy 2024-08-09T14:53:52.574 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.389+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:53:52.574 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.389+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:53:52.574 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.389+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:53:52.574 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.389+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=122}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:53:52.574 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.389+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c0047d0 con 0x55a86d4e4550 2024-08-09T14:53:52.574 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 28 ==== osd_map(122..126 src has 1..126) ==== 1675+0+0 (secure 0 0 0) 0x7fc3bc07e030 con 0x55a86d4642e0 2024-08-09T14:53:52.574 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(122..126 src has 1..126) 2024-08-09T14:53:52.574 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [122,126] > 121 2024-08-09T14:53:52.575 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 122 2024-08-09T14:53:52.575 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:52.575 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 122 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:52.575 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:52.575 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 123 2024-08-09T14:53:52.575 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:52.575 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 123 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:52.575 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:52.576 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 124 2024-08-09T14:53:52.576 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:52.576 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 124 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:52.576 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:52.576 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 125 2024-08-09T14:53:52.576 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:52.576 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 125 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:52.576 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:52.576 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 126 2024-08-09T14:53:55.015 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:53:55.001+0000 12a65640 -1 osd.7 127 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:53:56.043 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:53:56.032+0000 12a65640 -1 osd.7 128 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:53:57.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:57.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 126 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:53:57.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:53:57.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:53:57.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:52.572+0000 7fc3ba7fc640 20 client.4435.objecter 9 20.d3a59201 osd.0 exist [write 0~5 in=5b] 2024-08-09T14:53:57.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:53.343+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:57.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:53.343+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:57.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:53.343+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:57.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:54.343+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:57.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:54.343+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:57.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:54.343+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:57.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:55.343+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:57.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:55.343+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:57.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:55.343+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:57.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:56.343+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:57.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:56.343+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:57.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:56.343+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:57.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.343+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:53:57.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.343+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:53:57.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.343+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:53:57.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.390+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:53:57.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.390+0000 7fc3cb426640 2 client.4435.objecter tid 9 on osd.0 is laggy 2024-08-09T14:53:57.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.390+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:53:57.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.390+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:53:57.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.390+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:53:57.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.390+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=127}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:53:57.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.390+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c0047d0 con 0x55a86d4e4550 2024-08-09T14:53:57.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 29 ==== osd_map(127..129 src has 1..129) ==== 1900+0+0 (secure 0 0 0) 0x7fc3bc082020 con 0x55a86d4642e0 2024-08-09T14:53:57.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(127..129 src has 1..129) 2024-08-09T14:53:57.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [127,129] > 126 2024-08-09T14:53:57.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 127 2024-08-09T14:53:57.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:53:57.397 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:53:57.374+0000 12a65640 -1 osd.7 129 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:53:58.154 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:04:33.397 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:58.158 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:04:33.401 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:58.184 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:04:33.428 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:58.215 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:04:33.462 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:58.221 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:04:33.468 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:58.233 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:04:33.476 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:58.399 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:04:33.642 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:58.458 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:04:33.704 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:58.474 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:04:33.717 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:53:59.400 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:53:59.383+0000 12a65640 -1 osd.2 129 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:4 33.16 33:681f27cd:::2.obj:head [write 0~186 in=186b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:54:01.231 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:01.217+0000 12a65640 -1 osd.7 132 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:01.270 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:54:01.251+0000 ea5d640 -1 osd.4 132 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4480.0:15 18.3 18:c4b4050e:test-rados-api-smithi155-41975-4::foo:head [watch watch cookie 94507551719760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) 2024-08-09T14:54:02.327 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T14:54:02.318+0000 ea5d640 -1 osd.4 132 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4480.0:15 18.3 18:c4b4050e:test-rados-api-smithi155-41975-4::foo:head [watch watch cookie 94507551719760] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105) 2024-08-09T14:54:02.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 127 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:02.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:02.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 128 2024-08-09T14:54:02.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:54:02.392 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 128 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:02.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:02.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 129 2024-08-09T14:54:02.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:54:02.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 129 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:02.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:02.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:54:02.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:57.393+0000 7fc3ba7fc640 20 client.4435.objecter 9 20.d3a59201 osd.0 exist [write 0~5 in=5b] 2024-08-09T14:54:02.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:58.343+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:02.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:58.343+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:02.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:58.343+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:02.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:59.343+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:02.393 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:59.343+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:02.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:53:59.343+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:02.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:00.343+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:02.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:00.343+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:02.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:00.343+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:02.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:00.343+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:54:02.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:01.344+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:02.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:01.344+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:02.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:01.344+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:02.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.344+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:02.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.344+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:02.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.344+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:02.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.391+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:54:02.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.391+0000 7fc3cb426640 2 client.4435.objecter tid 9 on osd.0 is laggy 2024-08-09T14:54:02.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.391+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:54:02.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.391+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:54:02.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.391+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:54:02.496 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.391+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=130}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:54:02.497 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.391+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c0047d0 con 0x55a86d4e4550 2024-08-09T14:54:02.497 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 30 ==== osd_map(130..133 src has 1..133) ==== 1872+0+0 (secure 0 0 0) 0x7fc3bc07e070 con 0x55a86d4642e0 2024-08-09T14:54:02.497 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(130..133 src has 1..133) 2024-08-09T14:54:02.497 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [130,133] > 129 2024-08-09T14:54:02.497 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 130 2024-08-09T14:54:02.497 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:54:02.497 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 130 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:02.497 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:02.497 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 131 2024-08-09T14:54:02.497 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:54:02.497 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 131 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:02.497 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:02.498 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 132 2024-08-09T14:54:02.498 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:54:02.498 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 132 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:02.498 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:02.498 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 133 2024-08-09T14:54:02.498 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 10 client.4435.objecter checking op 9 2024-08-09T14:54:02.498 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 133 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:02.498 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:02.498 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:54:02.498 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.394+0000 7fc3ba7fc640 20 client.4435.objecter 9 20.d3a59201 osd.0 exist [write 0~5 in=5b] 2024-08-09T14:54:02.498 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (46430 ms) 2024-08-09T14:54:02.498 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-08-09T14:54:02.498 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 12 ==== osd_op_reply(9 exist [write 0~5] v119'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:54:02.499 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(9 exist [write 0~5] v119'2 uv2 ondisk = 0) 2024-08-09T14:54:02.499 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:54:02.526 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:02.494+0000 12a65640 -1 osd.7 133 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:02.790 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:54:02.755+0000 12a65640 -1 osd.2 132 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:4 33.16 33:681f27cd:::2.obj:head [write 0~186 in=186b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:54:03.487 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:03.471+0000 12a65640 -1 osd.7 133 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:03.980 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:54:03.973+0000 12a65640 -1 osd.2 133 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:4 33.16 33:681f27cd:::2.obj:head [write 0~186 in=186b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:54:04.669 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:04.637+0000 12a65640 -1 osd.7 134 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:05.233 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:54:05.149+0000 12a65640 -1 osd.2 134 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:4 33.16 33:681f27cd:::2.obj:head [write 0~186 in=186b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:54:05.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 9 ondisk uv 2 in 20.1 attempt 0 2024-08-09T14:54:05.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 0 2024-08-09T14:54:05.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 9 2024-08-09T14:54:05.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 9 2024-08-09T14:54:05.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:05.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 9 2024-08-09T14:54:05.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3c99c3640 5 client.4435.objecter 0 in flight 2024-08-09T14:54:05.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3c99c3640 20 librados: complete_aio_write 0x55a86d376850 2024-08-09T14:54:05.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-08-09T14:54:05.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 20 librados: queue_aio_write 0x55a86d4f5580 completion 0x55a86d365c90 write_seq 2 2024-08-09T14:54:05.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d4e3b90 2024-08-09T14:54:05.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 133 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:05.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:05.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:54:05.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:54:05.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid exist '@20' '@20' [write 0~5 in=5b] tid 10 osd.0 2024-08-09T14:54:05.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:54:05.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 10 2024-08-09T14:54:05.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 10 to 20.1 on osd.0 2024-08-09T14:54:05.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:10 20.1 20:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133) -- 0x55a86d5aa2f0 con 0x55a86d4e4550 2024-08-09T14:54:05.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:05.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 5 client.4435.objecter 1 in flight 2024-08-09T14:54:05.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:02.494+0000 7fc3cb6c6780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-08-09T14:54:05.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:03.344+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:05.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:03.344+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:05.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:03.344+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:05.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:04.344+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:05.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:04.344+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:05.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:04.344+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:05.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:05.344+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:05.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:05.344+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:05.770 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:05.737+0000 12a65640 -1 osd.7 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:06.382 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:54:06.188+0000 12a65640 -1 osd.2 135 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:4 33.16 33:681f27cd:::2.obj:head [write 0~186 in=186b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:54:06.877 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:06.836+0000 12a65640 -1 osd.7 136 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:07.363 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:54:07.249+0000 12a65640 -1 osd.2 136 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:4 33.16 33:681f27cd:::2.obj:head [write 0~186 in=186b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:54:08.001 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:07.981+0000 12a65640 -1 osd.7 137 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:08.377 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:54:08.360+0000 12a65640 -1 osd.2 137 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:4 33.16 33:681f27cd:::2.obj:head [write 0~186 in=186b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:54:09.058 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:09.050+0000 12a65640 -1 osd.7 138 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:09.402 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:54:09.367+0000 12a65640 -1 osd.2 137 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:4 33.16 33:681f27cd:::2.obj:head [write 0~186 in=186b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:54:10.251 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:10.232+0000 12a65640 -1 osd.7 140 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:10.558 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:54:10.481+0000 12a65640 -1 osd.2 138 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:4 33.16 33:681f27cd:::2.obj:head [write 0~186 in=186b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:54:11.713 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:54:11.701+0000 12a65640 -1 osd.2 139 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:4 33.16 33:681f27cd:::2.obj:head [write 0~186 in=186b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:54:11.804 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:11.793+0000 12a65640 -1 osd.7 141 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:12.802 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:12.793+0000 12a65640 -1 osd.7 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:13.005 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T14:54:12.934+0000 12a65640 -1 osd.2 140 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4454.0:4 33.16 33:681f27cd:::2.obj:head [write 0~186 in=186b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) 2024-08-09T14:54:13.920 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:13.896+0000 12a65640 -1 osd.7 143 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:14.500 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:14.473+0000 ea5d640 -1 osd.5 141 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:2 82.1e 82.9682be5e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e89) 2024-08-09T14:54:15.927 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:15.879+0000 ea5d640 -1 osd.5 142 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:2 82.1e 82:7a7d4169:::src_object.1:head [writefull 0~4096 in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) 2024-08-09T14:54:16.028 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:16.017+0000 12a65640 -1 osd.7 145 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:17.291 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:17.243+0000 ea5d640 -1 osd.5 143 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:2 82.1e 82:7a7d4169:::src_object.1:head [writefull 0~4096 in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) 2024-08-09T14:54:17.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:05.344+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:17.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:06.344+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:17.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:06.344+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:17.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:06.344+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:17.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:07.344+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:17.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:07.344+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:17.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:07.344+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:17.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:07.392+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:54:17.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:08.344+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:17.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:08.344+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:17.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:08.344+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:17.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:09.344+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:17.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:09.344+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:17.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:09.344+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:17.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:10.344+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:17.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:10.344+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:17.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:10.344+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:17.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:10.344+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:54:17.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:11.344+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:17.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:11.344+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:17.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:11.344+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:17.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:12.345+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:17.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:12.345+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:17.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:12.345+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:17.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:12.393+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:54:17.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:13.346+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:17.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:13.346+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:17.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:13.346+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:17.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:14.345+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:17.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:14.345+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:17.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:14.345+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:17.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:15.345+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:17.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:15.345+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:17.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:15.345+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:17.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:16.345+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:17.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:16.345+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:17.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:16.345+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:17.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.345+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:17.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.345+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:17.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.345+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:17.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.393+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:54:17.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.393+0000 7fc3cb426640 2 client.4435.objecter tid 10 on osd.0 is laggy 2024-08-09T14:54:17.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.393+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:54:17.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.393+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:54:17.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.393+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:54:17.400 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.393+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=134}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:54:17.400 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.393+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c0047d0 con 0x55a86d4e4550 2024-08-09T14:54:17.400 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 31 ==== osd_map(134..148 src has 1..148) ==== 13395+0+0 (secure 0 0 0) 0x7fc3bc082020 con 0x55a86d4642e0 2024-08-09T14:54:17.400 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(134..148 src has 1..148) 2024-08-09T14:54:17.400 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [134,148] > 133 2024-08-09T14:54:17.400 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 134 2024-08-09T14:54:17.400 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:17.400 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 134 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:17.400 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:17.400 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 135 2024-08-09T14:54:17.400 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:17.401 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 135 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:17.401 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:17.401 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 136 2024-08-09T14:54:17.401 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:17.401 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 136 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:17.401 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:17.401 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 137 2024-08-09T14:54:17.401 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:17.401 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 137 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:17.401 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:17.401 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 138 2024-08-09T14:54:17.401 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:17.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 138 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:17.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:17.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 139 2024-08-09T14:54:17.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:17.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 139 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:17.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:17.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 140 2024-08-09T14:54:17.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:17.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 140 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:17.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:17.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 141 2024-08-09T14:54:17.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:17.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 141 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:17.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:17.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 142 2024-08-09T14:54:17.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:17.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 142 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:17.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:17.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 143 2024-08-09T14:54:17.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:17.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 143 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:17.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:17.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 144 2024-08-09T14:54:17.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:17.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 144 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:17.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:17.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 145 2024-08-09T14:54:17.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:17.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 145 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:17.572 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:17.493+0000 12a65640 -1 osd.7 147 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:18.667 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:18.579+0000 ea5d640 -1 osd.5 145 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:2 82.1e 82:7a7d4169:::src_object.1:head [writefull 0~4096 in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89) 2024-08-09T14:54:18.862 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:18.674+0000 12a65640 -1 osd.7 148 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84:ad551702:::test:head [write 0~4 in=4b,writefull 0~2 in=2b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:19.605 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:19.605 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 146 2024-08-09T14:54:19.605 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:19.606 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 146 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:19.606 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:19.606 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 147 2024-08-09T14:54:19.606 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:19.606 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 147 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:19.606 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:19.606 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 148 2024-08-09T14:54:19.606 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 10 client.4435.objecter checking op 10 2024-08-09T14:54:19.606 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 148 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:19.606 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:19.606 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:54:19.606 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:17.397+0000 7fc3ba7fc640 20 client.4435.objecter 10 20.d3a59201 osd.0 exist [write 0~5 in=5b] 2024-08-09T14:54:19.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:18.345+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:19.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:18.345+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:19.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:18.345+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:19.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.345+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:19.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.345+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:19.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.345+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:19.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (17109 ms) 2024-08-09T14:54:19.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-08-09T14:54:19.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.603+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 13 ==== osd_op_reply(10 exist [write 0~5] v133'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:54:19.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.603+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(10 exist [write 0~5] v133'3 uv3 ondisk = 0) 2024-08-09T14:54:19.607 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.603+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:54:19.608 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.603+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 10 ondisk uv 3 in 20.1 attempt 0 2024-08-09T14:54:19.608 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.603+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 0 2024-08-09T14:54:19.608 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.603+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 10 2024-08-09T14:54:19.608 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.603+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 10 2024-08-09T14:54:19.608 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.603+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:19.671 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:19.660+0000 12a65640 -1 osd.7 148 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84:ad551702:::test:head [write 0~4 in=4b,writefull 0~2 in=2b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:21.293 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:21.212+0000 12a65640 -1 osd.7 150 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84:ad551702:::test:head [write 0~4 in=4b,writefull 0~2 in=2b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:22.285 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.603+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 10 2024-08-09T14:54:22.285 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.603+0000 7fc3c99c3640 5 client.4435.objecter 0 in flight 2024-08-09T14:54:22.285 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.603+0000 7fc3c99c3640 20 librados: complete_aio_write 0x55a86d365c90 2024-08-09T14:54:22.286 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-08-09T14:54:22.286 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 20 librados: queue_aio_write 0x55a86d4f5580 completion 0x55a86d4e3b90 write_seq 3 2024-08-09T14:54:22.286 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d432060 2024-08-09T14:54:22.286 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 148 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:22.286 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:22.286 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:54:22.286 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:54:22.286 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid exist '@20' '@20' [write 0~5 in=5b] tid 11 osd.0 2024-08-09T14:54:22.286 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:54:22.286 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 11 2024-08-09T14:54:22.286 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 11 to 20.1 on osd.0 2024-08-09T14:54:22.287 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:11 20.1 20:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148) -- 0x55a86d432510 con 0x55a86d4e4550 2024-08-09T14:54:22.287 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:22.287 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 5 client.4435.objecter 1 in flight 2024-08-09T14:54:22.287 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:19.604+0000 7fc3cb6c6780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-08-09T14:54:22.287 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:20.345+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:22.287 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:20.345+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:22.287 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:20.345+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:22.287 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:20.345+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:54:22.287 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:21.345+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:22.287 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:21.345+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:22.287 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:21.345+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:22.287 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:22.284+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 14 ==== osd_map(149..149 src has 1..150) ==== 1051+0+0 (crc 0 0 0) 0x7fc3c00085b0 con 0x55a86d4e4550 2024-08-09T14:54:22.288 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:22.284+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(149..149 src has 1..150) 2024-08-09T14:54:22.288 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:22.284+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [149,149] > 148 2024-08-09T14:54:22.288 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:22.284+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 149 2024-08-09T14:54:22.288 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:22.284+0000 7fc3ba7fc640 10 client.4435.objecter checking op 11 2024-08-09T14:54:22.395 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:22.260+0000 12a65640 -1 osd.7 151 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84:ad551702:::test:head [write 0~4 in=4b,writefull 0~2 in=2b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:23.032 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:22.935+0000 ea5d640 -1 osd.5 150 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:17.596138+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:24.000 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:04:59.243 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:24.003 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:04:59.246 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:24.062 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:04:59.309 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:24.064 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:04:59.311 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:24.089 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:04:59.333 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:24.091 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:04:59.334 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:24.152 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:04:59.396 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:24.216 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:04:59.463 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:24.252 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:04:59.495 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:24.359 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:24.308+0000 12a65640 -1 osd.7 153 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84:ad551702:::test:head [write 0~4 in=4b,writefull 0~2 in=2b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:24.401 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:24.389+0000 ea5d640 -1 osd.5 151 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:17.596138+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:25.584 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:25.464+0000 ea5d640 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:17.596138+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:25.630 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:25.546+0000 12a65640 -1 osd.7 154 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84:ad551702:::test:head [write 0~4 in=4b,writefull 0~2 in=2b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:26.674 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:26.664+0000 ea5d640 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:26.649340+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:27.756 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:27.703+0000 ea5d640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:27.064320+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:28.488 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:28.445+0000 12a65640 -1 osd.7 157 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84:ad551702:::test:head [write 0~4 in=4b,writefull 0~2 in=2b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:28.800 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:28.706+0000 ea5d640 -1 osd.5 154 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:27.064320+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:30.052 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:29.904+0000 ea5d640 -1 osd.5 155 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:27.064320+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:30.730 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:30.725+0000 12a65640 -1 osd.7 159 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84:ad551702:::test:head [write 0~4 in=4b,writefull 0~2 in=2b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:31.171 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:31.104+0000 ea5d640 -1 osd.5 156 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:27.064320+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:31.912 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:31.875+0000 12a65640 -1 osd.7 160 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84:ad551702:::test:head [write 0~4 in=4b,writefull 0~2 in=2b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:32.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:22.284+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 149 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:32.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:22.284+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:32.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:22.284+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:54:32.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:22.284+0000 7fc3ba7fc640 20 client.4435.objecter 11 20.d3a59201 osd.0 exist [write 0~5 in=5b] 2024-08-09T14:54:32.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:22.345+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:32.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:22.345+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:32.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:22.345+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:32.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:22.394+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:54:32.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:23.345+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:32.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:23.345+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:32.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:23.346+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:32.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:24.346+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:32.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:24.346+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:32.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:24.346+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:32.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:25.346+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:32.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:25.346+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:32.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:25.346+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:32.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:26.346+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:32.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:26.346+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:32.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:26.346+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:32.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:27.346+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:32.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:27.346+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:32.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:27.346+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:32.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:27.395+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:54:32.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:28.346+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:32.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:28.346+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:32.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:28.346+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:32.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:29.346+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:32.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:29.346+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:32.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:29.346+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:32.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:30.346+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:32.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:30.346+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:32.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:30.346+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:32.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:30.346+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:54:32.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:31.346+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:32.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:31.346+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:32.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:31.346+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:32.353 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:32.310+0000 ea5d640 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:27.064320+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:32.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.346+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:32.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.346+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:32.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.346+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:32.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.396+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:54:32.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.396+0000 7fc3cb426640 2 client.4435.objecter tid 11 on osd.0 is laggy 2024-08-09T14:54:32.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.396+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:54:32.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.396+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:54:32.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.396+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:54:32.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.396+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=150}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:54:32.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.396+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004830 con 0x55a86d4e4550 2024-08-09T14:54:32.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 32 ==== osd_map(150..161 src has 1..161) ==== 10223+0+0 (secure 0 0 0) 0x7fc3bc082030 con 0x55a86d4642e0 2024-08-09T14:54:32.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(150..161 src has 1..161) 2024-08-09T14:54:32.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [150,161] > 149 2024-08-09T14:54:32.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 150 2024-08-09T14:54:32.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 10 client.4435.objecter checking op 11 2024-08-09T14:54:32.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 150 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:32.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:32.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 151 2024-08-09T14:54:32.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 10 client.4435.objecter checking op 11 2024-08-09T14:54:32.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 151 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:32.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:32.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 152 2024-08-09T14:54:32.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 10 client.4435.objecter checking op 11 2024-08-09T14:54:32.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 152 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:32.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:32.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 153 2024-08-09T14:54:32.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 10 client.4435.objecter checking op 11 2024-08-09T14:54:32.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 153 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:32.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:32.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 154 2024-08-09T14:54:32.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 10 client.4435.objecter checking op 11 2024-08-09T14:54:32.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 154 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:32.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:32.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 155 2024-08-09T14:54:32.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 10 client.4435.objecter checking op 11 2024-08-09T14:54:32.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 155 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:32.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:32.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 156 2024-08-09T14:54:32.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 10 client.4435.objecter checking op 11 2024-08-09T14:54:32.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 156 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:32.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:32.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 157 2024-08-09T14:54:32.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 10 client.4435.objecter checking op 11 2024-08-09T14:54:32.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 157 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:32.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:32.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 158 2024-08-09T14:54:32.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 10 client.4435.objecter checking op 11 2024-08-09T14:54:32.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 158 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:32.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:32.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 159 2024-08-09T14:54:32.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 10 client.4435.objecter checking op 11 2024-08-09T14:54:32.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 159 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:32.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:32.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 160 2024-08-09T14:54:32.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 10 client.4435.objecter checking op 11 2024-08-09T14:54:32.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 160 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:32.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:32.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 161 2024-08-09T14:54:33.300 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:33.249+0000 12a65640 -1 osd.7 161 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84:ad551702:::test:head [write 0~4 in=4b,writefull 0~2 in=2b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:33.915 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:33.798+0000 ea5d640 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:27.064320+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:34.026 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 10 client.4435.objecter checking op 11 2024-08-09T14:54:34.026 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 161 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:34.026 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:34.026 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:54:34.026 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:32.432+0000 7fc3ba7fc640 20 client.4435.objecter 11 20.d3a59201 osd.0 exist [write 0~5 in=5b] 2024-08-09T14:54:34.026 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:33.346+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:34.026 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:33.346+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:34.026 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:33.346+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:34.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 15 ==== osd_op_reply(11 exist [write 0~5] v149'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:54:34.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(11 exist [write 0~5] v149'4 uv4 ondisk = 0) 2024-08-09T14:54:34.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:54:34.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 11 ondisk uv 4 in 20.1 attempt 0 2024-08-09T14:54:34.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 0 2024-08-09T14:54:34.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 11 2024-08-09T14:54:34.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 11 2024-08-09T14:54:34.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:34.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 11 2024-08-09T14:54:34.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3c99c3640 5 client.4435.objecter 0 in flight 2024-08-09T14:54:34.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3c99c3640 20 librados: complete_aio_write 0x55a86d4e3b90 2024-08-09T14:54:34.027 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (14420 ms) 2024-08-09T14:54:34.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-08-09T14:54:34.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d4319f0 2024-08-09T14:54:34.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 161 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:54:34.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:34.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:54:34.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:54:34.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid exist '@20' '@20' [read 0~0] tid 12 osd.0 2024-08-09T14:54:34.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:54:34.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 12 2024-08-09T14:54:34.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 12 to 20.1 on osd.0 2024-08-09T14:54:34.617 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:54:34.532+0000 12a65640 -1 osd.7 162 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:2 84.15 84:ad551702:::test:head [write 0~4 in=4b,writefull 0~2 in=2b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) 2024-08-09T14:54:35.715 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:35.557+0000 ea5d640 -1 osd.5 161 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:34.396475+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:36.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:12 20.1 20:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e161) -- 0x55a86d431ea0 con 0x55a86d4e4550 2024-08-09T14:54:36.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:36.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 5 client.4435.objecter 1 in flight 2024-08-09T14:54:36.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d432240 2024-08-09T14:54:36.801 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 161 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:54:36.801 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2024-08-09T14:54:36.801 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.a7bbaa1 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:54:36.801 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:36.801 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid noexist '@20' '@20' [read 0~0] tid 13 osd.0 2024-08-09T14:54:36.801 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:36.801 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 13 2024-08-09T14:54:36.801 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 13 to 20.1 on osd.0 2024-08-09T14:54:36.801 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:13 20.1 20:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e161) -- 0x55a86d5a92e0 con 0x55a86d4e4550 2024-08-09T14:54:36.801 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 5 2024-08-09T14:54:36.801 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.024+0000 7fc3cb6c6780 5 client.4435.objecter 2 in flight 2024-08-09T14:54:36.802 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.346+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:36.802 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.346+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:36.802 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:34.346+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:36.802 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:35.347+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:36.802 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:35.347+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:36.802 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:35.347+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:36.802 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.347+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:36.802 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.347+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:36.802 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.347+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:36.802 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 16 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:54:36.803 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-08-09T14:54:36.803 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:54:36.803 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 12 ondisk uv 4 in 20.1 attempt 0 2024-08-09T14:54:36.803 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 5 2024-08-09T14:54:36.803 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 12 2024-08-09T14:54:36.803 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 12 2024-08-09T14:54:36.803 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 5 2024-08-09T14:54:36.803 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 12 2024-08-09T14:54:36.803 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 5 client.4435.objecter 1 in flight 2024-08-09T14:54:36.803 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 17 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:54:36.803 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-09T14:54:36.803 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:54:36.804 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 13 ondisk uv 0 in 20.1 attempt 0 2024-08-09T14:54:36.804 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 10 client.4435.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-09T14:54:36.804 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 0 2024-08-09T14:54:36.804 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 13 2024-08-09T14:54:36.804 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 13 2024-08-09T14:54:36.804 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:36.804 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 13 2024-08-09T14:54:36.804 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3c99c3640 5 client.4435.objecter 0 in flight 2024-08-09T14:54:36.804 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (2774 ms) 2024-08-09T14:54:36.804 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-08-09T14:54:36.804 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d431b00 2024-08-09T14:54:36.804 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 161 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:54:36.805 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:36.805 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:54:36.805 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:54:36.805 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid exist '@20' '@20' [read 0~0] tid 14 osd.0 2024-08-09T14:54:36.805 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:54:36.805 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 14 2024-08-09T14:54:36.805 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 14 to 20.1 on osd.0 2024-08-09T14:54:36.805 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:14 20.1 20:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e161) -- 0x55a86d431fb0 con 0x55a86d4e4550 2024-08-09T14:54:36.805 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:36.805 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 5 client.4435.objecter 1 in flight 2024-08-09T14:54:36.906 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:36.856+0000 ea5d640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:34.396475+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:37.165 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d5a92e0 2024-08-09T14:54:37.165 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 161 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:54:37.165 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2024-08-09T14:54:37.165 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.a7bbaa1 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:54:37.166 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:37.166 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid noexist '@20' '@20' [read 0~0] tid 15 osd.0 2024-08-09T14:54:37.166 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:37.166 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 15 2024-08-09T14:54:37.166 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 15 to 20.1 on osd.0 2024-08-09T14:54:37.166 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:15 20.1 20:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e161) -- 0x55a86d5a9790 con 0x55a86d4e4550 2024-08-09T14:54:37.166 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 5 2024-08-09T14:54:37.166 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:36.798+0000 7fc3cb6c6780 5 client.4435.objecter 2 in flight 2024-08-09T14:54:37.166 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 18 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:54:37.166 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-08-09T14:54:37.166 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:54:37.166 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 14 ondisk uv 4 in 20.1 attempt 0 2024-08-09T14:54:37.167 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 5 2024-08-09T14:54:37.167 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 14 2024-08-09T14:54:37.167 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 14 2024-08-09T14:54:37.167 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 5 2024-08-09T14:54:37.167 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 14 2024-08-09T14:54:37.167 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 5 client.4435.objecter 1 in flight 2024-08-09T14:54:37.167 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 19 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:54:37.167 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-09T14:54:37.167 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:54:37.167 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 15 ondisk uv 0 in 20.1 attempt 0 2024-08-09T14:54:37.167 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 10 client.4435.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-09T14:54:37.167 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 0 2024-08-09T14:54:37.168 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 15 2024-08-09T14:54:37.168 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 15 2024-08-09T14:54:37.168 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:37.168 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 15 2024-08-09T14:54:37.168 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.163+0000 7fc3c99c3640 5 client.4435.objecter 0 in flight 2024-08-09T14:54:37.168 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (366 ms) 2024-08-09T14:54:37.168 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-08-09T14:54:37.168 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d5a92e0 2024-08-09T14:54:37.168 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 161 base exist @20 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:54:37.168 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:37.168 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:54:37.168 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:54:37.169 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid exist '@20' '@20' [read 0~0] tid 16 osd.0 2024-08-09T14:54:37.169 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:54:37.169 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 16 2024-08-09T14:54:37.169 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 16 to 20.1 on osd.0 2024-08-09T14:54:37.169 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:16 20.1 20:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e161) -- 0x55a86d5a9790 con 0x55a86d4e4550 2024-08-09T14:54:37.169 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:37.169 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 5 client.4435.objecter 1 in flight 2024-08-09T14:54:37.169 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d5a9b70 2024-08-09T14:54:37.169 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 161 base noexist @20 precalc_pgid 0 pgid 0.0 is_read 2024-08-09T14:54:37.169 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target noexist @20 -> pgid 20.a7bbaa1 2024-08-09T14:54:37.169 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.a7bbaa1 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:54:37.169 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:37.169 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid noexist '@20' '@20' [read 0~0] tid 17 osd.0 2024-08-09T14:54:37.170 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:37.170 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 17 2024-08-09T14:54:37.170 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 17 to 20.1 on osd.0 2024-08-09T14:54:37.839 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:17 20.1 20:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e161) -- 0x55a86d432920 con 0x55a86d4e4550 2024-08-09T14:54:37.839 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 5 2024-08-09T14:54:37.840 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.164+0000 7fc3cb6c6780 5 client.4435.objecter 2 in flight 2024-08-09T14:54:37.840 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.347+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:37.840 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.347+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:37.840 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.347+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:37.840 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.397+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:54:37.840 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 20 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:54:37.840 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-08-09T14:54:37.840 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:54:37.840 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 16 ondisk uv 4 in 20.1 attempt 0 2024-08-09T14:54:37.840 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 5 2024-08-09T14:54:37.840 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 16 2024-08-09T14:54:37.841 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 16 2024-08-09T14:54:37.841 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 5 2024-08-09T14:54:37.841 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 16 2024-08-09T14:54:37.841 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 5 client.4435.objecter 1 in flight 2024-08-09T14:54:37.841 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 21 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:54:37.841 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-08-09T14:54:37.841 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:54:37.841 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 17 ondisk uv 0 in 20.1 attempt 0 2024-08-09T14:54:37.841 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 10 client.4435.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-08-09T14:54:37.841 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 0 2024-08-09T14:54:37.841 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 17 2024-08-09T14:54:37.841 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 17 2024-08-09T14:54:37.842 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:37.842 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 17 2024-08-09T14:54:37.842 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3c99c3640 5 client.4435.objecter 0 in flight 2024-08-09T14:54:37.842 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (674 ms) 2024-08-09T14:54:38.104 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:38.091+0000 ea5d640 -1 osd.5 163 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:34.396475+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:39.259 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:39.208+0000 ea5d640 -1 osd.5 164 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:38.789839+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:40.348 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-08-09T14:54:40.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3cb6c6780 20 librados: queue_aio_write 0x55a86d4f5580 completion 0x55a86d4e3d00 write_seq 4 2024-08-09T14:54:40.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d431b00 2024-08-09T14:54:40.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 161 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:40.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:40.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:54:40.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:54:40.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid exist '@20' '@20' [writefull 0~5 in=5b] tid 18 osd.0 2024-08-09T14:54:40.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:54:40.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 18 2024-08-09T14:54:40.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 18 to 20.1 on osd.0 2024-08-09T14:54:40.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:18 20.1 20:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e161) -- 0x55a86d5a9b70 con 0x55a86d4e4550 2024-08-09T14:54:40.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:54:40.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:37.838+0000 7fc3cb6c6780 5 client.4435.objecter 1 in flight 2024-08-09T14:54:40.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:38.347+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:40.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:38.347+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:40.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:38.347+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:40.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:38.452+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 22 ==== osd_map(162..162 src has 1..162) ==== 2435+0+0 (crc 0 0 0) 0x7fc3c00085b0 con 0x55a86d4e4550 2024-08-09T14:54:40.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:38.452+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(162..162 src has 1..162) 2024-08-09T14:54:40.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:38.452+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [162,162] > 161 2024-08-09T14:54:40.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:38.452+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 162 2024-08-09T14:54:40.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:38.453+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:40.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:38.453+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 162 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:40.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:38.453+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:40.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:38.453+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:54:40.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:38.453+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:54:40.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:39.347+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:40.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:39.347+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:40.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:39.347+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:40.449 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:40.360+0000 ea5d640 -1 osd.5 165 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:38.789839+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:41.755 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:41.649+0000 ea5d640 -1 osd.5 166 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:38.789839+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:42.823 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:42.765+0000 ea5d640 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:38.789839+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:44.090 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:05:19.334 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:44.152 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:05:19.395 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:44.166 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:44.017+0000 ea5d640 -1 osd.5 168 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:38.789839+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:44.227 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:19.473 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:44.237 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:19.483 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:44.240 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:05:19.483 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:44.242 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:05:19.486 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:44.301 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:05:19.544 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:44.356 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:05:19.599 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:44.366 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:19.613 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:54:45.192 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:45.082+0000 ea5d640 -1 osd.5 169 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:38.789839+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:46.679 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:46.669+0000 ea5d640 -1 osd.5 170 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:38.789839+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:47.870 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:47.810+0000 ea5d640 -1 osd.5 171 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:46.798247+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:48.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:40.347+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:48.349 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:40.347+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:48.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:40.347+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:48.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:40.347+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:54:48.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:41.347+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:48.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:41.347+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:48.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:41.347+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:48.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:42.347+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:48.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:42.347+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:48.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:42.347+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:48.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:42.398+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:54:48.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:43.347+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:48.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:43.347+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:48.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:43.347+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:48.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:44.347+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:48.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:44.347+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:48.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:44.347+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:48.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:45.347+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:48.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:45.347+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:48.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:45.347+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:48.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:46.348+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:48.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:46.348+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:48.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:46.348+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:48.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:47.348+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:48.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:47.348+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:48.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:47.348+0000 7fc3b97fa640 10 monclient: _check_auth_tickets getting new tickets! 2024-08-09T14:54:48.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:47.348+0000 7fc3b97fa640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:54:48.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:47.348+0000 7fc3b97fa640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3a8001d80 con 0x55a86d4642e0 2024-08-09T14:54:48.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:47.348+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:48.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:47.399+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:54:48.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:47.399+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7fc39c002f90 con 0x55a86d4642e0 2024-08-09T14:54:48.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:47.399+0000 7fc3ba7fc640 10 monclient: _finish_auth 0 2024-08-09T14:54:48.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:47.399+0000 7fc3ba7fc640 10 monclient: _check_auth_tickets 2024-08-09T14:54:48.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:47.399+0000 7fc3ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:48.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:48.348+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:48.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:48.348+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:49.008 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:48.928+0000 ea5d640 -1 osd.5 173 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:47.885016+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:50.109 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:50.080+0000 ea5d640 -1 osd.5 174 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:47.885016+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:51.471 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:51.345+0000 ea5d640 -1 osd.5 175 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:50.973001+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:52.406 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:48.348+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:52.406 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:49.348+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:52.406 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:49.348+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:52.406 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:49.348+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:52.406 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:50.348+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:52.406 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:50.348+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:52.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:50.348+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:52.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:50.348+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:54:52.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:51.348+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:52.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:51.348+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:52.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:51.348+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:52.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.348+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:52.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.348+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:52.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.348+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:52.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.400+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:54:52.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.400+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:54:52.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.400+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:54:52.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.400+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:54:52.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.400+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:54:52.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.400+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=163}) -- 0x7fc39c003560 con 0x55a86d4642e0 2024-08-09T14:54:52.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.400+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004830 con 0x55a86d4e4550 2024-08-09T14:54:52.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 34 ==== osd_map(163..178 src has 1..178) ==== 9153+0+0 (secure 0 0 0) 0x7fc3bc036060 con 0x55a86d4642e0 2024-08-09T14:54:52.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(163..178 src has 1..178) 2024-08-09T14:54:52.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [163,178] > 162 2024-08-09T14:54:52.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 163 2024-08-09T14:54:52.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:52.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 163 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:52.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:52.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 164 2024-08-09T14:54:52.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:52.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 164 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:52.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:52.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 165 2024-08-09T14:54:52.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:52.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 165 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:52.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:52.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 166 2024-08-09T14:54:52.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:52.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 166 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:52.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:52.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 167 2024-08-09T14:54:52.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:52.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 167 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:52.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:52.410 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 168 2024-08-09T14:54:52.410 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:52.410 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 168 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:52.410 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:52.410 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 169 2024-08-09T14:54:52.410 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:52.410 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 169 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:52.410 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:52.410 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 170 2024-08-09T14:54:52.410 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:52.410 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 170 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:52.410 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:52.411 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 171 2024-08-09T14:54:52.411 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:52.411 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 171 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:52.411 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:52.850 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:52.845+0000 ea5d640 -1 osd.5 177 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:50.973001+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:53.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 172 2024-08-09T14:54:53.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:53.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 172 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:53.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:53.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 173 2024-08-09T14:54:53.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:53.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 173 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:53.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:53.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.404+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 174 2024-08-09T14:54:53.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:53.350 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 174 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:53.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:53.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 175 2024-08-09T14:54:53.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:53.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 175 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:53.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:53.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 176 2024-08-09T14:54:53.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:53.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 176 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:53.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:53.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 177 2024-08-09T14:54:53.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:53.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 177 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:53.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:53.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 178 2024-08-09T14:54:53.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:53.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 178 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:53.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:53.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:54:53.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:52.405+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:54:53.928 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:53.908+0000 ea5d640 -1 osd.5 178 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:50.973001+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:55.416 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:55.333+0000 ea5d640 -1 osd.5 179 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:50.973001+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:56.734 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:56.700+0000 ea5d640 -1 osd.5 180 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:55.738561+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:57.524 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:53.348+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:57.524 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:53.348+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:57.524 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:53.348+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:57.524 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:54.348+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:57.524 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:54.348+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:57.525 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:54.348+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:57.525 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:55.348+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:57.525 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:55.348+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:57.525 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:55.348+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:57.525 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:56.348+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:57.525 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:56.348+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:57.525 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:56.348+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:57.525 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.349+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:54:57.525 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.349+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:54:57.525 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.349+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:54:57.525 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.401+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:54:57.525 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.401+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:54:57.526 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.401+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:54:57.526 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.401+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:54:57.526 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.401+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:54:57.526 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.401+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=179}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:54:57.526 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.401+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004a00 con 0x55a86d4e4550 2024-08-09T14:54:57.526 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 35 ==== osd_map(179..183 src has 1..183) ==== 3594+0+0 (secure 0 0 0) 0x7fc3bc082020 con 0x55a86d4642e0 2024-08-09T14:54:57.526 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(179..183 src has 1..183) 2024-08-09T14:54:57.526 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [179,183] > 178 2024-08-09T14:54:57.526 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 179 2024-08-09T14:54:57.526 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:57.526 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 179 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:54:57.526 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:54:57.526 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 180 2024-08-09T14:54:57.527 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:54:57.934 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:57.900+0000 ea5d640 -1 osd.5 181 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:56.890954+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:54:59.586 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:54:59.528+0000 ea5d640 -1 osd.5 182 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:56.890954+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:00.613 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:00.592+0000 ea5d640 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:56.890954+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:01.635 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:01.627+0000 ea5d640 -1 osd.5 184 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:54:56.890954+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:02.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 180 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:02.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:02.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 181 2024-08-09T14:55:02.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:02.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 181 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:02.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:02.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 182 2024-08-09T14:55:02.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:02.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 182 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:02.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:02.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 183 2024-08-09T14:55:02.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:02.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 183 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:02.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:02.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:55:02.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:57.522+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:55:02.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:58.349+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:02.405 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:58.349+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:02.405 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:58.349+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:02.405 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:59.349+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:02.405 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:59.349+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:02.405 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:54:59.349+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:02.405 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:00.349+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:02.405 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:00.349+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:02.405 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:00.349+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:02.405 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:00.349+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:55:02.405 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:01.349+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:02.405 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:01.349+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:02.405 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:01.349+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:02.405 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.349+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:02.406 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.349+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:02.406 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.349+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:02.406 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.402+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:55:02.406 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.402+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:55:02.819 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:02.764+0000 ea5d640 -1 osd.5 186 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:01.725401+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:03.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.402+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:55:03.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.402+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:55:03.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.402+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:55:03.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.402+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=184}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:55:03.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.402+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:55:03.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 36 ==== osd_map(184..187 src has 1..187) ==== 2118+0+0 (secure 0 0 0) 0x7fc3bc082020 con 0x55a86d4642e0 2024-08-09T14:55:03.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(184..187 src has 1..187) 2024-08-09T14:55:03.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [184,187] > 183 2024-08-09T14:55:03.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 184 2024-08-09T14:55:03.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:03.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 184 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:03.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:03.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 185 2024-08-09T14:55:03.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:03.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 185 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:03.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:03.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 186 2024-08-09T14:55:03.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:03.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 186 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:03.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:03.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 187 2024-08-09T14:55:03.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:03.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 187 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:03.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:03.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:55:03.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:02.405+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:55:03.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:03.350+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:03.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:03.350+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:04.029 INFO:tasks.ceph.mon.a.smithi155.stderr:2024-08-09T14:55:04.021+0000 d7c3640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.6 [v2:172.21.15.194:6808/3077539270,v1:172.21.15.194:6809/3077539270] for 84sec e151 v151) 2024-08-09T14:55:04.035 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:05:39.279 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:04.101 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:39.347 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:04.102 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:05:39.346 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:04.109 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:39.356 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:04.109 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:04.041+0000 ea5d640 -1 osd.5 186 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:01.725401+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:04.171 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:05:39.414 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:04.177 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:05:39.420 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:04.214 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:39.461 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:04.288 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:05:39.532 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:04.396 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:05:39.639 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:05.076 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:05.066+0000 ea5d640 -1 osd.5 187 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:01.725401+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:05.829 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:05.771+0000 12a65640 -1 osd.0 186 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4837.0:7 90.1s0 90:98181865:ns1::foo6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137) 2024-08-09T14:55:05.983 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:05.885+0000 ea5d640 -1 osd.3 188 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:06.298 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:06.128+0000 ea5d640 -1 osd.5 188 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:05.956112+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:07.013 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:06.912+0000 12a65640 -1 osd.0 187 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4837.0:7 90.1s0 90:98181865:ns1::foo6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137) 2024-08-09T14:55:07.031 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:06.923+0000 ea5d640 -1 osd.3 189 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:07.274 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:07.159+0000 ea5d640 -1 osd.5 189 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:05.956112+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:07.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:03.350+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:07.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:04.349+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:07.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:04.349+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:07.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:04.349+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:07.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:05.349+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:07.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:05.349+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:07.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:05.349+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:07.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:06.349+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:07.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:06.349+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:07.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:06.349+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:07.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.349+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:07.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.349+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:07.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.349+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:07.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.402+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:55:07.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.402+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:55:07.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.402+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:55:07.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.402+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:55:07.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.402+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:55:07.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.402+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=188}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:55:07.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.402+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:55:07.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 37 ==== osd_map(188..191 src has 1..191) ==== 1843+0+0 (secure 0 0 0) 0x7fc3bc07e070 con 0x55a86d4642e0 2024-08-09T14:55:07.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(188..191 src has 1..191) 2024-08-09T14:55:07.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [188,191] > 187 2024-08-09T14:55:07.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 188 2024-08-09T14:55:07.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:07.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 188 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:07.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:07.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 189 2024-08-09T14:55:07.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:07.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 189 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:07.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:08.150 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:07.964+0000 12a65640 -1 osd.0 188 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4837.0:7 90.1s0 90:98181865:ns1::foo6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137) 2024-08-09T14:55:08.207 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:08.085+0000 ea5d640 -1 osd.3 190 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:08.288 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:08.208+0000 ea5d640 -1 osd.5 190 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:05.956112+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:09.067 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:55:09.058+0000 12a65640 -1 osd.7 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:4 84.15 84:ad551702:::test:head [writefull 0~4 [fadvise_nocache] in=4b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163) 2024-08-09T14:55:09.299 INFO:tasks.ceph.mon.a.smithi155.stderr:2024-08-09T14:55:09.297+0000 d7c3640 -1 mon.a@0(leader) e1 get_health_metrics reporting 1 slow ops, oldest is osd_failure(failed timeout osd.6 [v2:172.21.15.194:6808/3077539270,v1:172.21.15.194:6809/3077539270] for 84sec e151 v151) 2024-08-09T14:55:09.496 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:09.423+0000 12a65640 -1 osd.0 189 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4837.0:7 90.1s0 90:98181865:ns1::foo6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137) 2024-08-09T14:55:09.501 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:09.361+0000 ea5d640 -1 osd.3 191 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:09.683 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:09.453+0000 ea5d640 -1 osd.5 191 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:05.956112+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:10.556 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:55:10.513+0000 12a65640 -1 osd.7 193 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:4 84.15 84:ad551702:::test:head [writefull 0~4 [fadvise_nocache] in=4b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163) 2024-08-09T14:55:10.887 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:10.868+0000 ea5d640 -1 osd.5 192 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:09.467782+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:10.906 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:10.895+0000 12a65640 -1 osd.0 190 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4837.0:7 90.1s0 90:98181865:ns1::foo6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137) 2024-08-09T14:55:11.700 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:55:11.682+0000 12a65640 -1 osd.7 194 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:4 84.15 84:ad551702:::test:head [writefull 0~4 [fadvise_nocache] in=4b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163) 2024-08-09T14:55:12.039 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:12.010+0000 12a65640 -1 osd.0 191 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4837.0:7 90.1s0 90:98181865:ns1::foo6:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137) 2024-08-09T14:55:12.180 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:12.036+0000 ea5d640 -1 osd.5 193 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:09.467782+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:12.476 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 190 2024-08-09T14:55:12.476 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:12.476 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 190 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:12.476 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:12.476 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 191 2024-08-09T14:55:12.476 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:12.476 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 191 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:12.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:12.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:55:12.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:07.405+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:55:12.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:08.350+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:12.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:08.350+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:12.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:08.350+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:12.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:09.350+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:12.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:09.350+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:12.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:09.350+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:12.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:10.350+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:12.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:10.350+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:12.477 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:10.350+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:12.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:10.350+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:55:12.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:11.350+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:12.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:11.350+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:12.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:11.350+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:12.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.350+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:12.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.350+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:12.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.350+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:12.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.403+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:55:12.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.403+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:55:12.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.403+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:55:12.478 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.403+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:55:12.479 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.403+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:55:12.479 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.403+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=192}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:55:12.479 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.403+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:55:12.971 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:55:12.903+0000 12a65640 -1 osd.7 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:4 84.15 84:ad551702:::test:head [writefull 0~4 [fadvise_nocache] in=4b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163) 2024-08-09T14:55:13.137 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:13.117+0000 ea5d640 -1 osd.5 194 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:09.467782+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:13.140 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:13.130+0000 ea5d640 -1 osd.5 194 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100.5b267ac7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:14.055 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:14.003+0000 ea5d640 -1 osd.3 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:14.058 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:55:13.992+0000 12a65640 -1 osd.7 196 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:4 84.15 84:ad551702:::test:head [writefull 0~4 [fadvise_nocache] in=4b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163) 2024-08-09T14:55:14.233 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:14.122+0000 ea5d640 -1 osd.5 195 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:09.467782+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:14.351 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 38 ==== osd_map(192..196 src has 1..196) ==== 1744+0+0 (secure 0 0 0) 0x7fc3bc07e060 con 0x55a86d4642e0 2024-08-09T14:55:14.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(192..196 src has 1..196) 2024-08-09T14:55:14.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [192,196] > 191 2024-08-09T14:55:14.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 192 2024-08-09T14:55:14.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:14.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 192 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:14.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:14.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 193 2024-08-09T14:55:14.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:14.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 193 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:14.352 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:14.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 194 2024-08-09T14:55:14.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:14.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 194 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:14.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:14.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 195 2024-08-09T14:55:14.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:14.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 195 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:14.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:14.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 196 2024-08-09T14:55:14.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:14.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 196 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:14.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:14.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:55:14.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:12.474+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:55:14.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:13.350+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:14.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:13.350+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:14.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:13.350+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:14.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:14.350+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:14.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:14.350+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:14.389 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:14.280+0000 ea5d640 -1 osd.5 195 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100.5b267ac7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:15.063 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:55:15.056+0000 12a65640 -1 osd.7 196 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:4 84.15 84:ad551702:::test:head [writefull 0~4 [fadvise_nocache] in=4b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163) 2024-08-09T14:55:15.373 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:15.269+0000 ea5d640 -1 osd.3 196 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:15.486 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:15.307+0000 ea5d640 -1 osd.5 196 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:09.467782+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:15.544 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:15.470+0000 ea5d640 -1 osd.5 196 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100.5b267ac7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:16.386 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:16.372+0000 ea5d640 -1 osd.3 196 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:16.610 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T14:55:16.594+0000 12a65640 -1 osd.7 198 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4819.0:4 84.15 84:ad551702:::test:head [writefull 0~4 [fadvise_nocache] in=4b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e163) 2024-08-09T14:55:16.617 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:16.499+0000 ea5d640 -1 osd.5 196 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:16.223482+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:16.646 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:16.601+0000 ea5d640 -1 osd.5 196 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100.5b267ac7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:17.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:14.350+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:17.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:15.350+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:17.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:15.350+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:17.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:15.350+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:17.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:16.350+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:17.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:16.350+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:17.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:16.350+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:17.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.350+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:17.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.350+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:17.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.350+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:17.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.404+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:55:17.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.404+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:55:17.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.404+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:55:17.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.404+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:55:17.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.404+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:55:17.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.404+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=197}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:55:17.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.404+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:55:17.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 39 ==== osd_map(197..201 src has 1..201) ==== 4320+0+0 (secure 0 0 0) 0x7fc3bc082030 con 0x55a86d4642e0 2024-08-09T14:55:17.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(197..201 src has 1..201) 2024-08-09T14:55:17.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [197,201] > 196 2024-08-09T14:55:17.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 197 2024-08-09T14:55:17.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:17.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 197 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:17.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:17.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 198 2024-08-09T14:55:17.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:17.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 198 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:17.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:17.438 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 199 2024-08-09T14:55:17.439 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:17.925 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:17.910+0000 ea5d640 -1 osd.3 199 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:18.522 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:18.289+0000 ea5d640 -1 osd.5 198 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:16.223482+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:18.655 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:18.609+0000 ea5d640 -1 osd.5 199 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:18.971 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 199 base exist @20 precalc_p api_watch_notify_pp: Running main() from gmock_main.cc 2024-08-09T14:55:18.971 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-08-09T14:55:18.971 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-08-09T14:55:18.972 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-08-09T14:55:18.972 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-08-09T14:55:18.972 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: notify 2024-08-09T14:55:18.972 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (40806 ms) 2024-08-09T14:55:18.972 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-08-09T14:55:18.972 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (28373 ms) 2024-08-09T14:55:18.972 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (69180 ms total) 2024-08-09T14:55:18.972 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: 2024-08-09T14:55:18.972 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-08-09T14:55:18.972 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-08-09T14:55:18.972 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: notify 2024-08-09T14:55:18.972 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (28719 ms) 2024-08-09T14:55:18.973 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-08-09T14:55:18.973 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: notify 2024-08-09T14:55:18.973 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (21409 ms) 2024-08-09T14:55:18.973 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-08-09T14:55:18.973 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (13853 ms) 2024-08-09T14:55:18.973 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-08-09T14:55:18.973 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (8749 ms) 2024-08-09T14:55:18.973 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-08-09T14:55:18.973 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: handle_notify cookie 94471279659024 notify_id 670014898179 notifier_gid 4804 2024-08-09T14:55:18.973 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: handle_notify cookie 94471279659024 notify_id 674309865476 notifier_gid 4804 2024-08-09T14:55:18.973 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (9957 ms) 2024-08-09T14:55:18.973 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-08-09T14:55:18.973 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: handle_notify cookie 94471279659024 notify_id 704374636549 notifier_gid 4804 2024-08-09T14:55:18.974 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (11641 ms) 2024-08-09T14:55:18.974 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-08-09T14:55:18.974 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: handle_notify cookie 94471279659024 notify_id 743029342214 notifier_gid 4804 2024-08-09T14:55:18.974 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: handle_notify cookie 94471279659024 notify_id 747324309511 notifier_gid 4804 2024-08-09T14:55:18.974 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (7552 ms) 2024-08-09T14:55:18.974 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-08-09T14:55:18.974 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: handle_notify cookie 94471279659024 notify_id 764504178696 notifier_gid 4804 2024-08-09T14:55:18.974 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (6239 ms) 2024-08-09T14:55:18.974 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-08-09T14:55:18.974 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: handle_notify cookie 94471279659024 notify_id 807453851657 notifier_gid 4804 2024-08-09T14:55:18.974 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (11337 ms) 2024-08-09T14:55:18.974 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-08-09T14:55:18.975 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: handle_notify cookie 94471279659024 notify_id 854698491914 notifier_gid 4804 2024-08-09T14:55:18.975 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (12084 ms) 2024-08-09T14:55:19.009 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:18.981+0000 ea5d640 -1 osd.3 199 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:20.873 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:20.797+0000 ea5d640 -1 osd.5 200 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:16.223482+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:21.034 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:21.017+0000 ea5d640 -1 osd.5 200 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:21.244 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:21.200+0000 ea5d640 -1 osd.3 201 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:22.137 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:22.016+0000 ea5d640 -1 osd.5 203 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:16.223482+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:22.406 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timgid 0 pgid 0.0 is_write 2024-08-09T14:55:22.406 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:22.406 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 200 2024-08-09T14:55:22.406 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:22.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 200 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:22.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:22.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 201 2024-08-09T14:55:22.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:22.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 201 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:22.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:22.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:55:22.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:17.434+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:55:22.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:18.350+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:22.407 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:18.350+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:22.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:18.350+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:22.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:19.350+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:22.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:19.350+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:22.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:19.350+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:22.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:20.351+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:22.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:20.351+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:22.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:20.351+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:22.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:20.351+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:55:22.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:21.351+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:22.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:21.351+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:22.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:21.351+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:22.408 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.351+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:22.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.351+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:22.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.351+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:22.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.405+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:55:22.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.405+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:55:22.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.405+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:55:22.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.405+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:55:22.409 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.405+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:55:22.497 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:22.484+0000 ea5d640 -1 osd.3 203 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:22.789 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.405+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=202}) -- 0x7fc39c004830 con 0x55a86d4642e api_stat: Running main() from gmock_main.cc 2024-08-09T14:55:22.790 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-08-09T14:55:22.790 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [----------] Global test environment set-up. 2024-08-09T14:55:22.790 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-08-09T14:55:22.790 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-08-09T14:55:22.790 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ OK ] LibRadosStat.Stat (57590 ms) 2024-08-09T14:55:22.790 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-08-09T14:55:22.790 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (19371 ms) 2024-08-09T14:55:22.790 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-08-09T14:55:22.790 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ OK ] LibRadosStat.StatNS (16777 ms) 2024-08-09T14:55:22.791 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-08-09T14:55:22.791 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (17 ms) 2024-08-09T14:55:22.791 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-08-09T14:55:22.791 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (7442 ms) 2024-08-09T14:55:22.791 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [----------] 5 tests from LibRadosStat (101197 ms total) 2024-08-09T14:55:22.791 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: 2024-08-09T14:55:22.791 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-08-09T14:55:22.791 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-08-09T14:55:22.791 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (52241 ms) 2024-08-09T14:55:22.791 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-08-09T14:55:22.791 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (39395 ms) 2024-08-09T14:55:22.791 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-08-09T14:55:22.791 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (49 ms) 2024-08-09T14:55:22.792 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-08-09T14:55:22.792 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (10459 ms) 2024-08-09T14:55:22.792 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (102144 ms total) 2024-08-09T14:55:22.792 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: 2024-08-09T14:55:22.792 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [----------] Global test environment tear-down 2024-08-09T14:55:22.792 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (215426 ms total) 2024-08-09T14:55:22.792 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat: [ PASSED ] 9 tests. 2024-08-09T14:55:22.825 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:22.754+0000 ea5d640 -1 osd.5 203 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:23.721 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:23.590+0000 ea5d640 -1 osd.3 204 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:23.834 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:23.786+0000 ea5d640 -1 osd.5 204 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:16.223482+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:23.849 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:23.801+0000 ea5d640 -1 osd.5 204 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:24.655 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:05:59.899 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:24.709 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:05:59.952 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:24.719 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:59.966 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:24.726 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:05:59.973 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:24.743 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:05:59.987 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:24.753 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:05:59.997 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:24.754 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:24.715+0000 ea5d640 -1 osd.3 205 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:24.815 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:00.062 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:24.873 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:00.116 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:24.901 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:00.144 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:24.938 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:24.896+0000 ea5d640 -1 osd.5 205 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:16.223482+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:25.248 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:25.168+0000 ea5d640 -1 osd.5 205 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:25.352 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-08-09T14:55:25.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.405+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:55:25.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 40 ==== osd_map(202..205 src has 1..205) ==== 1132+0+0 (secure 0 0 0) 0x7fc3bc082070 con 0x55a86d4642e0 2024-08-09T14:55:25.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(202..205 src has 1..205) 2024-08-09T14:55:25.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [202,205] > 201 2024-08-09T14:55:25.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 202 2024-08-09T14:55:25.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:25.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 202 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:25.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:25.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 203 2024-08-09T14:55:25.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:25.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 203 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:25.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:25.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 204 2024-08-09T14:55:25.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:25.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 204 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:25.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:25.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 205 2024-08-09T14:55:25.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:25.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 205 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:25.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:25.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:55:25.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:22.408+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:55:25.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:23.351+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:25.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:23.351+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:25.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:23.351+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:25.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:24.351+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:25.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:24.351+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:25.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:24.351+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:25.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:25.351+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:25.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:25.351+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:25.821 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:25.793+0000 ea5d640 -1 osd.3 206 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:26.771 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:26.610+0000 ea5d640 -1 osd.5 207 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:53:00.221599+0000 front 2024-08-09T14:55:16.223482+0000 (oldest deadline 2024-08-09T14:54:22.639319+0000) 2024-08-09T14:55:26.774 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:26.676+0000 ea5d640 -1 osd.5 207 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:26.973 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:26.959+0000 ea5d640 -1 osd.3 207 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:27.411 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:25.351+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:27.411 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:26.351+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:27.411 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:26.351+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:27.411 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:26.351+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:27.411 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.351+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:27.411 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.351+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:27.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.351+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:27.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.406+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:55:27.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.406+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:55:27.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.406+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:55:27.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.406+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:55:27.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.406+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:55:27.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.406+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=206}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:55:27.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.406+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:55:27.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.409+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 41 ==== osd_map(206..210 src has 1..210) ==== 4251+0+0 (secure 0 0 0) 0x7fc3bc082020 con 0x55a86d4642e0 2024-08-09T14:55:27.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.409+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(206..210 src has 1..210) 2024-08-09T14:55:27.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.409+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [206,210] > 205 2024-08-09T14:55:27.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.409+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 206 2024-08-09T14:55:27.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.409+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:27.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.409+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 206 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:27.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.409+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:27.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.409+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 207 2024-08-09T14:55:27.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:27.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 207 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:27.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:27.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 208 2024-08-09T14:55:27.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:27.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 208 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:27.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:27.755 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:27.742+0000 ea5d640 -1 osd.5 207 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:28.144 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:28.136+0000 ea5d640 -1 osd.3 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:29.188 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:29.153+0000 ea5d640 -1 osd.5 209 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:29.680 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:29.566+0000 ea5d640 -1 osd.3 209 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:30.217 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:30.208+0000 ea5d640 -1 osd.5 209 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:31.293 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:31.276+0000 ea5d640 -1 osd.5 211 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:31.696 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:31.667+0000 ea5d640 -1 osd.3 211 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:32.411 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 209 2024-08-09T14:55:32.411 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:32.411 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 209 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:32.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:32.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 210 2024-08-09T14:55:32.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:32.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 210 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:32.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:32.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:55:32.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:27.410+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:55:32.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:28.351+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:32.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:28.351+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:32.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:28.351+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:32.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:29.351+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:32.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:29.351+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:32.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:29.351+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:32.412 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:30.351+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:32.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:30.351+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:32.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:30.351+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:32.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:30.351+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:55:32.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:31.352+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:32.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:31.352+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:32.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:31.352+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:32.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.352+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:32.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.352+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:32.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.352+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:32.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.407+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:55:32.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.407+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:55:32.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.407+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:55:32.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.407+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:55:32.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.407+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:55:32.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.407+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=211}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:55:32.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.407+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:55:32.769 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:32.754+0000 ea5d640 -1 osd.5 212 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:33.508 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:33.486+0000 ea5d640 -1 osd.3 212 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:33.946 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:33.909+0000 ea5d640 -1 osd.5 214 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:34.626 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:34.538+0000 ea5d640 -1 osd.3 213 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:34.952 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:34.945+0000 ea5d640 -1 osd.5 215 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:36.007 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:35.862+0000 ea5d640 -1 osd.3 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:36.353 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 42 ==== osd_map(211..214 src has 1..214) ==== 1455+0+0 (secure 0 0 0) 0x7fc3bc082070 con 0x55a86d4642e0 2024-08-09T14:55:36.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(211..214 src has 1..214) 2024-08-09T14:55:36.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [211,214] > 210 2024-08-09T14:55:36.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 211 2024-08-09T14:55:36.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:36.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 211 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:36.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:36.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 212 2024-08-09T14:55:36.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:36.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 212 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:36.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:36.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 213 2024-08-09T14:55:36.354 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:36.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 213 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:36.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:36.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 214 2024-08-09T14:55:36.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:36.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 214 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:36.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:36.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:55:36.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:32.410+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:55:36.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:33.352+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:36.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:33.352+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:36.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:33.352+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:36.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:34.352+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:36.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:34.352+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:36.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:34.352+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:36.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:35.352+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:36.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:35.352+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:36.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:35.352+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:36.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:36.352+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:36.861 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:36.787+0000 ea5d640 -1 osd.5 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:37.115 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:37.000+0000 ea5d640 -1 osd.3 216 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:37.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:36.352+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:37.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:36.352+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:37.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.352+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:37.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.352+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:37.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.352+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:37.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.408+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:55:37.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.408+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:55:37.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.408+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:55:37.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.408+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:55:37.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.408+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:55:37.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.408+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=215}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:55:37.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.408+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:55:37.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 43 ==== osd_map(215..219 src has 1..219) ==== 2472+0+0 (secure 0 0 0) 0x7fc3bc082020 con 0x55a86d4642e0 2024-08-09T14:55:37.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(215..219 src has 1..219) 2024-08-09T14:55:37.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [215,219] > 214 2024-08-09T14:55:37.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 215 2024-08-09T14:55:37.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:37.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 215 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:37.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:37.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 216 2024-08-09T14:55:37.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:37.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 216 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:37.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:37.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 217 2024-08-09T14:55:37.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:37.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 217 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:37.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:37.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 218 2024-08-09T14:55:37.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:37.811 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:37.786+0000 ea5d640 -1 osd.5 217 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:38.388 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:38.363+0000 ea5d640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:39.001 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:38.889+0000 ea5d640 -1 osd.5 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:39.463 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:39.381+0000 ea5d640 -1 osd.3 218 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4706.0:5 82.2 82:41780a1c:::tgt_object:head [call test_remote_reads.test_gather in=233b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e157) 2024-08-09T14:55:40.134 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:40.077+0000 ea5d640 -1 osd.5 219 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:41.197 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:41.187+0000 ea5d640 -1 osd.5 220 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:42.363 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:42.327+0000 ea5d640 -1 osd.5 221 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4760.0:1 100.7s0 100:e35e64da:test-rados-api-smithi155-41823-8::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e151) 2024-08-09T14:55:42.430 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 218 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:42.430 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:42.430 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 219 2024-08-09T14:55:42.430 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:42.430 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 219 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:42.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:42.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:55:42.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:37.430+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:55:42.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:38.352+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:42.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:38.352+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:42.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:38.352+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:42.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:39.352+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:42.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:39.352+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:42.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:39.352+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:42.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:40.352+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:42.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:40.352+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:42.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:40.352+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:42.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:40.352+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:55:42.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:41.352+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:42.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:41.352+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:42.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:41.352+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:42.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.353+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:42.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.353+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:42.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.353+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:42.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.409+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:55:42.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.409+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:55:42.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.409+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:55:42.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.409+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:55:42.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.409+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:55:42.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.409+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=220}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:55:42.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.409+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:55:42.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 44 ==== osd_map(220..224 src has 1..224) ==== 5425+0+0 (secure 0 0 0) 0x7fc3bc07e060 con 0x55a86d4642e0 2024-08-09T14:55:43.526 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:43.473+0000 12a65640 -1 osd.0 217 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4700.0:7 80.18 80:1c3bf51a:test-rados-api-smithi155-41609-4::ws:head [write-same in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-08-09T14:55:44.663 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:44.607+0000 12a65640 -1 osd.0 218 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4700.0:7 80.18 80:1c3bf51a:test-rados-api-smithi155-41609-4::ws:head [write-same in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-08-09T14:55:44.855 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:20.098 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:44.933 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:20.179 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:44.937 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:20.183 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:44.945 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:20.188 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:45.023 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:20.266 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:45.025 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:20.268 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:45.044 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:20.291 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:45.065 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:20.308 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:45.144 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:20.387 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:55:45.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(220..224 src has 1..224) 2024-08-09T14:55:45.355 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [220,224] > 219 2024-08-09T14:55:45.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 220 2024-08-09T14:55:45.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:45.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 220 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:45.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:45.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 221 2024-08-09T14:55:45.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:45.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 221 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:45.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:45.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 222 2024-08-09T14:55:45.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:45.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 222 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:45.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:45.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 223 2024-08-09T14:55:45.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:45.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 223 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:45.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:45.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.428+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 224 2024-08-09T14:55:45.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.429+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:45.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.429+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 224 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:45.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.429+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:45.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.429+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:55:45.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:42.429+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:55:45.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:43.353+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:45.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:43.353+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:45.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:43.353+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:45.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:44.353+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:45.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:44.353+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:45.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:44.353+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:45.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:45.354+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:45.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:45.354+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:47.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:45.354+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:47.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:46.354+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:47.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:46.354+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:47.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:46.354+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:47.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.354+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:47.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.354+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:47.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.354+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:47.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.410+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:55:47.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.410+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:55:47.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.410+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:55:47.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.410+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:55:47.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.410+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:55:47.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.410+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=225}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:55:47.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.410+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:55:47.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 45 ==== osd_map(225..228 src has 1..228) ==== 1851+0+0 (secure 0 0 0) 0x7fc3bc082020 con 0x55a86d4642e0 2024-08-09T14:55:47.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(225..228 src has 1..228) 2024-08-09T14:55:47.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [225,228] > 224 2024-08-09T14:55:47.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 225 2024-08-09T14:55:47.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:47.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 225 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:47.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:47.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 226 2024-08-09T14:55:47.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:47.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 226 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:47.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:47.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 227 2024-08-09T14:55:47.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:47.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 227 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:47.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:47.467 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:47.416+0000 12a65640 -1 osd.0 221 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4700.0:7 80.18 80:1c3bf51a:test-rados-api-smithi155-41609-4::ws:head [write-same in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-08-09T14:55:48.659 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:48.562+0000 12a65640 -1 osd.0 221 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4700.0:7 80.18 80:1c3bf51a:test-rados-api-smithi155-41609-4::ws:head [write-same in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-08-09T14:55:50.160 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:50.069+0000 12a65640 -1 osd.0 223 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4700.0:7 80.18 80:1c3bf51a:test-rados-api-smithi155-41609-4::ws:head [write-same in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-08-09T14:55:52.039 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:52.033+0000 12a65640 -1 osd.0 224 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4700.0:7 80.18 80:1c3bf51a:test-rados-api-smithi155-41609-4::ws:head [write-same in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-08-09T14:55:52.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 228 2024-08-09T14:55:52.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:52.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 228 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:52.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:52.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:55:52.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:47.412+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:55:52.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:48.354+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:52.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:48.354+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:52.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:48.354+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:52.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:49.354+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:52.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:49.354+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:52.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:49.354+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:52.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:50.354+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:52.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:50.354+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:52.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:50.354+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:52.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:50.354+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:55:52.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:51.354+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:52.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:51.354+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:52.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:51.354+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:52.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.354+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:52.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.354+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:52.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.354+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:52.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.412+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:55:52.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.412+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:55:52.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.412+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:55:52.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.412+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:55:52.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.412+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:55:52.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.412+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=229}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:55:52.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.412+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:55:52.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.413+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 46 ==== osd_map(229..232 src has 1..232) ==== 1068+0+0 (secure 0 0 0) 0x7fc3bc082020 con 0x55a86d4642e0 2024-08-09T14:55:52.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.413+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(229..232 src has 1..232) 2024-08-09T14:55:52.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.413+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [229,232] > 228 2024-08-09T14:55:54.672 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:54.657+0000 12a65640 -1 osd.0 226 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4700.0:7 80.18 80:1c3bf51a:test-rados-api-smithi155-41609-4::ws:head [write-same in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-08-09T14:55:55.806 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:55.796+0000 ea5d640 -1 osd.5 233 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4671.0:3 102.17 102:e96903aa:test-rados-api-smithi155-41701-8::foo:head [writefull 0~4 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165) 2024-08-09T14:55:55.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:55.766+0000 12a65640 -1 osd.0 228 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:55:57.379 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:57.372+0000 12a65640 -1 osd.0 230 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:55:57.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.413+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 229 2024-08-09T14:55:57.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.413+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:57.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.413+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 229 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:57.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.413+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:57.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.414+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 230 2024-08-09T14:55:57.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.414+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:57.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.414+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 230 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:57.413 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.414+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:57.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.414+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 231 2024-08-09T14:55:57.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.414+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:57.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.414+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 231 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:57.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.414+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:57.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.414+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 232 2024-08-09T14:55:57.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.414+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:57.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.414+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 232 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:57.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.414+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:57.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.414+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:55:57.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:52.414+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:55:57.414 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:53.356+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:57.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:53.356+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:57.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:53.356+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:57.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:54.355+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:57.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:54.355+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:57.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:54.355+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:57.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:55.355+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:57.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:55.355+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:57.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:55.355+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:57.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:56.355+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:57.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:56.355+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:57.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:56.355+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:57.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.355+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:57.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.355+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:55:57.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.355+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:55:57.683 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:57.629+0000 ea5d640 -1 osd.5 235 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4671.0:3 102.17 102:e96903aa:test-rados-api-smithi155-41701-8::foo:head [writefull 0~4 in=4b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e165) 2024-08-09T14:55:58.356 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.412+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:55:58.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.412+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:55:58.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.412+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:55:58.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.412+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:55:58.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.412+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:55:58.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.412+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=233}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:55:58.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.412+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:55:58.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 47 ==== osd_map(233..236 src has 1..236) ==== 1448+0+0 (secure 0 0 0) 0x7fc3bc036070 con 0x55a86d4642e0 2024-08-09T14:55:58.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(233..236 src has 1..236) 2024-08-09T14:55:58.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [233,236] > 232 2024-08-09T14:55:58.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 233 2024-08-09T14:55:58.357 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:58.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 233 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:58.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:58.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 234 2024-08-09T14:55:58.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:58.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 234 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:58.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:58.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 235 2024-08-09T14:55:58.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:58.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 235 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:58.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:58.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 236 2024-08-09T14:55:58.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:55:58.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 236 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:55:58.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:55:58.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:55:58.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:57.415+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:55:58.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:58.355+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:55:58.667 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:58.657+0000 12a65640 -1 osd.0 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:55:59.124 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:55:59.111+0000 ea5d640 -1 osd.5 237 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4479.0:27 27.11 27:8e16cba4:::25.obj:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206) 2024-08-09T14:55:59.768 INFO:tasks.ceph.osd.3.smithi155.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1345-g0a88632b/rpm/el9/BUILD/ceph-19.1.0-1345-g0a88632b/src/osd/PrimaryLogPG.cc: In function 'int PrimaryLogPG::start_cls_gather(PrimaryLogPG::OpContext*, std::map, ceph::buffer::v15_2_0::list>*, const string&, const char*, const char*, ceph::bufferlist&)' thread 25a91640 time 2024-08-09T14:55:59.762143+0000 2024-08-09T14:55:59.769 INFO:tasks.ceph.osd.3.smithi155.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1345-g0a88632b/rpm/el9/BUILD/ceph-19.1.0-1345-g0a88632b/src/osd/PrimaryLogPG.cc: 10387: FAILED ceph_assert(inserted) 2024-08-09T14:55:59.934 INFO:tasks.ceph.osd.3.smithi155.stderr: ceph version 19.1.0-1345-g0a88632b (0a88632b45f25ae38493667c7dbd155f6e59b499) squid (rc) 2024-08-09T14:55:59.934 INFO:tasks.ceph.osd.3.smithi155.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x5004c0] 2024-08-09T14:55:59.934 INFO:tasks.ceph.osd.3.smithi155.stderr: 2: ceph-osd(+0x3f867c) [0x50067c] 2024-08-09T14:55:59.934 INFO:tasks.ceph.osd.3.smithi155.stderr: 3: ceph-osd(+0x3976e0) [0x49f6e0] 2024-08-09T14:55:59.934 INFO:tasks.ceph.osd.3.smithi155.stderr: 4: (cls_cxx_gather(void*, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > > const&, std::__cxx11::basic_string, std::allocator > const&, char const*, char const*, ceph::buffer::v15_2_0::list&)+0x34a) [0x5f9fba] 2024-08-09T14:55:59.934 INFO:tasks.ceph.osd.3.smithi155.stderr: 5: /usr/lib64/rados-classes/libcls_test_remote_reads.so(+0x3861) [0x167c1861] 2024-08-09T14:55:59.934 INFO:tasks.ceph.osd.3.smithi155.stderr: 6: ceph-osd(+0x5e380d) [0x6eb80d] 2024-08-09T14:55:59.934 INFO:tasks.ceph.osd.3.smithi155.stderr: 7: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&)+0x5d) [0x6ec4bd] 2024-08-09T14:55:59.935 INFO:tasks.ceph.osd.3.smithi155.stderr: 8: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x594) [0x778ce4] 2024-08-09T14:55:59.935 INFO:tasks.ceph.osd.3.smithi155.stderr: 9: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x9e) [0x78f82e] 2024-08-09T14:55:59.935 INFO:tasks.ceph.osd.3.smithi155.stderr: 10: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x37e) [0x768b4e] 2024-08-09T14:55:59.935 INFO:tasks.ceph.osd.3.smithi155.stderr: 11: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x342b) [0x75fd3b] 2024-08-09T14:55:59.935 INFO:tasks.ceph.osd.3.smithi155.stderr: 12: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x69bed7] 2024-08-09T14:55:59.935 INFO:tasks.ceph.osd.3.smithi155.stderr: 13: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x8e6879] 2024-08-09T14:55:59.935 INFO:tasks.ceph.osd.3.smithi155.stderr: 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x6b6177] 2024-08-09T14:55:59.935 INFO:tasks.ceph.osd.3.smithi155.stderr: 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0xbb44ba] 2024-08-09T14:55:59.935 INFO:tasks.ceph.osd.3.smithi155.stderr: 16: ceph-osd(+0xaaca64) [0xbb4a64] 2024-08-09T14:55:59.935 INFO:tasks.ceph.osd.3.smithi155.stderr: 17: /lib64/libc.so.6(+0x89c02) [0x54dbc02] 2024-08-09T14:55:59.935 INFO:tasks.ceph.osd.3.smithi155.stderr: 18: clone() 2024-08-09T14:55:59.981 INFO:tasks.ceph.osd.3.smithi155.stderr:*** Caught signal (Aborted) ** 2024-08-09T14:55:59.981 INFO:tasks.ceph.osd.3.smithi155.stderr: in thread 25a91640 thread_name:tp_osd_tp 2024-08-09T14:56:00.009 INFO:tasks.ceph.osd.3.smithi155.stderr: ceph version 19.1.0-1345-g0a88632b (0a88632b45f25ae38493667c7dbd155f6e59b499) squid (rc) 2024-08-09T14:56:00.009 INFO:tasks.ceph.osd.3.smithi155.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x54906f0] 2024-08-09T14:56:00.009 INFO:tasks.ceph.osd.3.smithi155.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x54dd94c] 2024-08-09T14:56:00.009 INFO:tasks.ceph.osd.3.smithi155.stderr: 3: raise() 2024-08-09T14:56:00.009 INFO:tasks.ceph.osd.3.smithi155.stderr: 4: abort() 2024-08-09T14:56:00.009 INFO:tasks.ceph.osd.3.smithi155.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x50051b] 2024-08-09T14:56:00.009 INFO:tasks.ceph.osd.3.smithi155.stderr: 6: ceph-osd(+0x3f867c) [0x50067c] 2024-08-09T14:56:00.009 INFO:tasks.ceph.osd.3.smithi155.stderr: 7: ceph-osd(+0x3976e0) [0x49f6e0] 2024-08-09T14:56:00.010 INFO:tasks.ceph.osd.3.smithi155.stderr: 8: (cls_cxx_gather(void*, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > > const&, std::__cxx11::basic_string, std::allocator > const&, char const*, char const*, ceph::buffer::v15_2_0::list&)+0x34a) [0x5f9fba] 2024-08-09T14:56:00.010 INFO:tasks.ceph.osd.3.smithi155.stderr: 9: /usr/lib64/rados-classes/libcls_test_remote_reads.so(+0x3861) [0x167c1861] 2024-08-09T14:56:00.010 INFO:tasks.ceph.osd.3.smithi155.stderr: 10: ceph-osd(+0x5e380d) [0x6eb80d] 2024-08-09T14:56:00.010 INFO:tasks.ceph.osd.3.smithi155.stderr: 11: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&)+0x5d) [0x6ec4bd] 2024-08-09T14:56:00.010 INFO:tasks.ceph.osd.3.smithi155.stderr: 12: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x594) [0x778ce4] 2024-08-09T14:56:00.010 INFO:tasks.ceph.osd.3.smithi155.stderr: 13: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x9e) [0x78f82e] 2024-08-09T14:56:00.010 INFO:tasks.ceph.osd.3.smithi155.stderr: 14: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x37e) [0x768b4e] 2024-08-09T14:56:00.010 INFO:tasks.ceph.osd.3.smithi155.stderr: 15: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x342b) [0x75fd3b] 2024-08-09T14:56:00.010 INFO:tasks.ceph.osd.3.smithi155.stderr: 16: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x69bed7] 2024-08-09T14:56:00.010 INFO:tasks.ceph.osd.3.smithi155.stderr: 17: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x8e6879] 2024-08-09T14:56:00.010 INFO:tasks.ceph.osd.3.smithi155.stderr: 18: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x6b6177] 2024-08-09T14:56:00.010 INFO:tasks.ceph.osd.3.smithi155.stderr: 19: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0xbb44ba] 2024-08-09T14:56:00.011 INFO:tasks.ceph.osd.3.smithi155.stderr: 20: ceph-osd(+0xaaca64) [0xbb4a64] 2024-08-09T14:56:00.011 INFO:tasks.ceph.osd.3.smithi155.stderr: 21: /lib64/libc.so.6(+0x89c02) [0x54dbc02] 2024-08-09T14:56:00.011 INFO:tasks.ceph.osd.3.smithi155.stderr: 22: clone() 2024-08-09T14:56:00.038 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:55:59.933+0000 25a91640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1345-g0a88632b/rpm/el9/BUILD/ceph-19.1.0-1345-g0a88632b/src/osd/PrimaryLogPG.cc: In function 'int PrimaryLogPG::start_cls_gather(PrimaryLogPG::OpContext*, std::map, ceph::buffer::v15_2_0::list>*, const string&, const char*, const char*, ceph::bufferlist&)' thread 25a91640 time 2024-08-09T14:55:59.762143+0000 2024-08-09T14:56:00.038 INFO:tasks.ceph.osd.3.smithi155.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1345-g0a88632b/rpm/el9/BUILD/ceph-19.1.0-1345-g0a88632b/src/osd/PrimaryLogPG.cc: 10387: FAILED ceph_assert(inserted) 2024-08-09T14:56:00.038 INFO:tasks.ceph.osd.3.smithi155.stderr: 2024-08-09T14:56:00.038 INFO:tasks.ceph.osd.3.smithi155.stderr: ceph version 19.1.0-1345-g0a88632b (0a88632b45f25ae38493667c7dbd155f6e59b499) squid (rc) 2024-08-09T14:56:00.039 INFO:tasks.ceph.osd.3.smithi155.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x5004c0] 2024-08-09T14:56:00.039 INFO:tasks.ceph.osd.3.smithi155.stderr: 2: ceph-osd(+0x3f867c) [0x50067c] 2024-08-09T14:56:00.039 INFO:tasks.ceph.osd.3.smithi155.stderr: 3: ceph-osd(+0x3976e0) [0x49f6e0] 2024-08-09T14:56:00.039 INFO:tasks.ceph.osd.3.smithi155.stderr: 4: (cls_cxx_gather(void*, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > > const&, std::__cxx11::basic_string, std::allocator > const&, char const*, char const*, ceph::buffer::v15_2_0::list&)+0x34a) [0x5f9fba] 2024-08-09T14:56:00.039 INFO:tasks.ceph.osd.3.smithi155.stderr: 5: /usr/lib64/rados-classes/libcls_test_remote_reads.so(+0x3861) [0x167c1861] 2024-08-09T14:56:00.039 INFO:tasks.ceph.osd.3.smithi155.stderr: 6: ceph-osd(+0x5e380d) [0x6eb80d] 2024-08-09T14:56:00.039 INFO:tasks.ceph.osd.3.smithi155.stderr: 7: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&)+0x5d) [0x6ec4bd] 2024-08-09T14:56:00.039 INFO:tasks.ceph.osd.3.smithi155.stderr: 8: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x594) [0x778ce4] 2024-08-09T14:56:00.039 INFO:tasks.ceph.osd.3.smithi155.stderr: 9: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x9e) [0x78f82e] 2024-08-09T14:56:00.039 INFO:tasks.ceph.osd.3.smithi155.stderr: 10: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x37e) [0x768b4e] 2024-08-09T14:56:00.039 INFO:tasks.ceph.osd.3.smithi155.stderr: 11: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x342b) [0x75fd3b] 2024-08-09T14:56:00.039 INFO:tasks.ceph.osd.3.smithi155.stderr: 12: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x69bed7] 2024-08-09T14:56:00.039 INFO:tasks.ceph.osd.3.smithi155.stderr: 13: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x8e6879] 2024-08-09T14:56:00.039 INFO:tasks.ceph.osd.3.smithi155.stderr: 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x6b6177] 2024-08-09T14:56:00.040 INFO:tasks.ceph.osd.3.smithi155.stderr: 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0xbb44ba] 2024-08-09T14:56:00.040 INFO:tasks.ceph.osd.3.smithi155.stderr: 16: ceph-osd(+0xaaca64) [0xbb4a64] 2024-08-09T14:56:00.040 INFO:tasks.ceph.osd.3.smithi155.stderr: 17: /lib64/libc.so.6(+0x89c02) [0x54dbc02] 2024-08-09T14:56:00.040 INFO:tasks.ceph.osd.3.smithi155.stderr: 18: clone() 2024-08-09T14:56:00.040 INFO:tasks.ceph.osd.3.smithi155.stderr: 2024-08-09T14:56:00.072 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:55:59.967+0000 12a65640 -1 osd.0 232 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:00.077 INFO:tasks.ceph.osd.3.smithi155.stderr:2024-08-09T14:56:00.038+0000 25a91640 -1 *** Caught signal (Aborted) ** 2024-08-09T14:56:00.077 INFO:tasks.ceph.osd.3.smithi155.stderr: in thread 25a91640 thread_name:tp_osd_tp 2024-08-09T14:56:00.077 INFO:tasks.ceph.osd.3.smithi155.stderr: 2024-08-09T14:56:00.077 INFO:tasks.ceph.osd.3.smithi155.stderr: ceph version 19.1.0-1345-g0a88632b (0a88632b45f25ae38493667c7dbd155f6e59b499) squid (rc) 2024-08-09T14:56:00.077 INFO:tasks.ceph.osd.3.smithi155.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x54906f0] 2024-08-09T14:56:00.077 INFO:tasks.ceph.osd.3.smithi155.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x54dd94c] 2024-08-09T14:56:00.077 INFO:tasks.ceph.osd.3.smithi155.stderr: 3: raise() 2024-08-09T14:56:00.077 INFO:tasks.ceph.osd.3.smithi155.stderr: 4: abort() 2024-08-09T14:56:00.077 INFO:tasks.ceph.osd.3.smithi155.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x50051b] 2024-08-09T14:56:00.077 INFO:tasks.ceph.osd.3.smithi155.stderr: 6: ceph-osd(+0x3f867c) [0x50067c] 2024-08-09T14:56:00.077 INFO:tasks.ceph.osd.3.smithi155.stderr: 7: ceph-osd(+0x3976e0) [0x49f6e0] 2024-08-09T14:56:00.077 INFO:tasks.ceph.osd.3.smithi155.stderr: 8: (cls_cxx_gather(void*, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > > const&, std::__cxx11::basic_string, std::allocator > const&, char const*, char const*, ceph::buffer::v15_2_0::list&)+0x34a) [0x5f9fba] 2024-08-09T14:56:00.078 INFO:tasks.ceph.osd.3.smithi155.stderr: 9: /usr/lib64/rados-classes/libcls_test_remote_reads.so(+0x3861) [0x167c1861] 2024-08-09T14:56:00.078 INFO:tasks.ceph.osd.3.smithi155.stderr: 10: ceph-osd(+0x5e380d) [0x6eb80d] 2024-08-09T14:56:00.078 INFO:tasks.ceph.osd.3.smithi155.stderr: 11: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&)+0x5d) [0x6ec4bd] 2024-08-09T14:56:00.078 INFO:tasks.ceph.osd.3.smithi155.stderr: 12: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x594) [0x778ce4] 2024-08-09T14:56:00.078 INFO:tasks.ceph.osd.3.smithi155.stderr: 13: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x9e) [0x78f82e] 2024-08-09T14:56:00.078 INFO:tasks.ceph.osd.3.smithi155.stderr: 14: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x37e) [0x768b4e] 2024-08-09T14:56:00.078 INFO:tasks.ceph.osd.3.smithi155.stderr: 15: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x342b) [0x75fd3b] 2024-08-09T14:56:00.078 INFO:tasks.ceph.osd.3.smithi155.stderr: 16: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x69bed7] 2024-08-09T14:56:00.078 INFO:tasks.ceph.osd.3.smithi155.stderr: 17: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x8e6879] 2024-08-09T14:56:00.078 INFO:tasks.ceph.osd.3.smithi155.stderr: 18: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x6b6177] 2024-08-09T14:56:00.078 INFO:tasks.ceph.osd.3.smithi155.stderr: 19: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0xbb44ba] 2024-08-09T14:56:00.078 INFO:tasks.ceph.osd.3.smithi155.stderr: 20: ceph-osd(+0xaaca64) [0xbb4a64] 2024-08-09T14:56:00.078 INFO:tasks.ceph.osd.3.smithi155.stderr: 21: /lib64/libc.so.6(+0x89c02) [0x54dbc02] 2024-08-09T14:56:00.078 INFO:tasks.ceph.osd.3.smithi155.stderr: 22: clone() 2024-08-09T14:56:00.079 INFO:tasks.ceph.osd.3.smithi155.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-08-09T14:56:00.079 INFO:tasks.ceph.osd.3.smithi155.stderr: 2024-08-09T14:56:00.418 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:00.267+0000 ea5d640 -1 osd.5 238 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4479.0:27 27.11 27:8e16cba4:::25.obj:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206) 2024-08-09T14:56:00.543 INFO:tasks.ceph.osd.3.smithi155.stderr: -1052> 2024-08-09T14:55:59.933+0000 25a91640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1345-g0a88632b/rpm/el9/BUILD/ceph-19.1.0-1345-g0a88632b/src/osd/PrimaryLogPG.cc: In function 'int PrimaryLogPG::start_cls_gather(PrimaryLogPG::OpContext*, std::map, ceph::buffer::v15_2_0::list>*, const string&, const char*, const char*, ceph::bufferlist&)' thread 25a91640 time 2024-08-09T14:55:59.762143+0000 2024-08-09T14:56:00.543 INFO:tasks.ceph.osd.3.smithi155.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1345-g0a88632b/rpm/el9/BUILD/ceph-19.1.0-1345-g0a88632b/src/osd/PrimaryLogPG.cc: 10387: FAILED ceph_assert(inserted) 2024-08-09T14:56:00.543 INFO:tasks.ceph.osd.3.smithi155.stderr: 2024-08-09T14:56:00.544 INFO:tasks.ceph.osd.3.smithi155.stderr: ceph version 19.1.0-1345-g0a88632b (0a88632b45f25ae38493667c7dbd155f6e59b499) squid (rc) 2024-08-09T14:56:00.544 INFO:tasks.ceph.osd.3.smithi155.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x5004c0] 2024-08-09T14:56:00.544 INFO:tasks.ceph.osd.3.smithi155.stderr: 2: ceph-osd(+0x3f867c) [0x50067c] 2024-08-09T14:56:00.544 INFO:tasks.ceph.osd.3.smithi155.stderr: 3: ceph-osd(+0x3976e0) [0x49f6e0] 2024-08-09T14:56:00.544 INFO:tasks.ceph.osd.3.smithi155.stderr: 4: (cls_cxx_gather(void*, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > > const&, std::__cxx11::basic_string, std::allocator > const&, char const*, char const*, ceph::buffer::v15_2_0::list&)+0x34a) [0x5f9fba] 2024-08-09T14:56:00.544 INFO:tasks.ceph.osd.3.smithi155.stderr: 5: /usr/lib64/rados-classes/libcls_test_remote_reads.so(+0x3861) [0x167c1861] 2024-08-09T14:56:00.544 INFO:tasks.ceph.osd.3.smithi155.stderr: 6: ceph-osd(+0x5e380d) [0x6eb80d] 2024-08-09T14:56:00.544 INFO:tasks.ceph.osd.3.smithi155.stderr: 7: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&)+0x5d) [0x6ec4bd] 2024-08-09T14:56:00.544 INFO:tasks.ceph.osd.3.smithi155.stderr: 8: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x594) [0x778ce4] 2024-08-09T14:56:00.544 INFO:tasks.ceph.osd.3.smithi155.stderr: 9: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x9e) [0x78f82e] 2024-08-09T14:56:00.544 INFO:tasks.ceph.osd.3.smithi155.stderr: 10: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x37e) [0x768b4e] 2024-08-09T14:56:00.544 INFO:tasks.ceph.osd.3.smithi155.stderr: 11: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x342b) [0x75fd3b] 2024-08-09T14:56:00.545 INFO:tasks.ceph.osd.3.smithi155.stderr: 12: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x69bed7] 2024-08-09T14:56:00.545 INFO:tasks.ceph.osd.3.smithi155.stderr: 13: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x8e6879] 2024-08-09T14:56:00.545 INFO:tasks.ceph.osd.3.smithi155.stderr: 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x6b6177] 2024-08-09T14:56:00.545 INFO:tasks.ceph.osd.3.smithi155.stderr: 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0xbb44ba] 2024-08-09T14:56:00.545 INFO:tasks.ceph.osd.3.smithi155.stderr: 16: ceph-osd(+0xaaca64) [0xbb4a64] 2024-08-09T14:56:00.545 INFO:tasks.ceph.osd.3.smithi155.stderr: 17: /lib64/libc.so.6(+0x89c02) [0x54dbc02] 2024-08-09T14:56:00.545 INFO:tasks.ceph.osd.3.smithi155.stderr: 18: clone() 2024-08-09T14:56:00.545 INFO:tasks.ceph.osd.3.smithi155.stderr: 2024-08-09T14:56:00.546 INFO:tasks.ceph.osd.3.smithi155.stderr: -938> 2024-08-09T14:56:00.038+0000 25a91640 -1 *** Caught signal (Aborted) ** 2024-08-09T14:56:00.546 INFO:tasks.ceph.osd.3.smithi155.stderr: in thread 25a91640 thread_name:tp_osd_tp 2024-08-09T14:56:00.547 INFO:tasks.ceph.osd.3.smithi155.stderr: 2024-08-09T14:56:00.547 INFO:tasks.ceph.osd.3.smithi155.stderr: ceph version 19.1.0-1345-g0a88632b (0a88632b45f25ae38493667c7dbd155f6e59b499) squid (rc) 2024-08-09T14:56:00.547 INFO:tasks.ceph.osd.3.smithi155.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x54906f0] 2024-08-09T14:56:00.547 INFO:tasks.ceph.osd.3.smithi155.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x54dd94c] 2024-08-09T14:56:00.547 INFO:tasks.ceph.osd.3.smithi155.stderr: 3: raise() 2024-08-09T14:56:00.547 INFO:tasks.ceph.osd.3.smithi155.stderr: 4: abort() 2024-08-09T14:56:00.547 INFO:tasks.ceph.osd.3.smithi155.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x50051b] 2024-08-09T14:56:00.547 INFO:tasks.ceph.osd.3.smithi155.stderr: 6: ceph-osd(+0x3f867c) [0x50067c] 2024-08-09T14:56:00.547 INFO:tasks.ceph.osd.3.smithi155.stderr: 7: ceph-osd(+0x3976e0) [0x49f6e0] 2024-08-09T14:56:00.547 INFO:tasks.ceph.osd.3.smithi155.stderr: 8: (cls_cxx_gather(void*, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > > const&, std::__cxx11::basic_string, std::allocator > const&, char const*, char const*, ceph::buffer::v15_2_0::list&)+0x34a) [0x5f9fba] 2024-08-09T14:56:00.547 INFO:tasks.ceph.osd.3.smithi155.stderr: 9: /usr/lib64/rados-classes/libcls_test_remote_reads.so(+0x3861) [0x167c1861] 2024-08-09T14:56:00.548 INFO:tasks.ceph.osd.3.smithi155.stderr: 10: ceph-osd(+0x5e380d) [0x6eb80d] 2024-08-09T14:56:00.548 INFO:tasks.ceph.osd.3.smithi155.stderr: 11: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&)+0x5d) [0x6ec4bd] 2024-08-09T14:56:00.548 INFO:tasks.ceph.osd.3.smithi155.stderr: 12: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x594) [0x778ce4] 2024-08-09T14:56:00.548 INFO:tasks.ceph.osd.3.smithi155.stderr: 13: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x9e) [0x78f82e] 2024-08-09T14:56:00.548 INFO:tasks.ceph.osd.3.smithi155.stderr: 14: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x37e) [0x768b4e] 2024-08-09T14:56:00.548 INFO:tasks.ceph.osd.3.smithi155.stderr: 15: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x342b) [0x75fd3b] 2024-08-09T14:56:00.548 INFO:tasks.ceph.osd.3.smithi155.stderr: 16: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x69bed7] 2024-08-09T14:56:00.548 INFO:tasks.ceph.osd.3.smithi155.stderr: 17: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x8e6879] 2024-08-09T14:56:00.548 INFO:tasks.ceph.osd.3.smithi155.stderr: 18: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x6b6177] 2024-08-09T14:56:00.548 INFO:tasks.ceph.osd.3.smithi155.stderr: 19: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0xbb44ba] 2024-08-09T14:56:00.548 INFO:tasks.ceph.osd.3.smithi155.stderr: 20: ceph-osd(+0xaaca64) [0xbb4a64] 2024-08-09T14:56:00.548 INFO:tasks.ceph.osd.3.smithi155.stderr: 21: /lib64/libc.so.6(+0x89c02) [0x54dbc02] 2024-08-09T14:56:00.548 INFO:tasks.ceph.osd.3.smithi155.stderr: 22: clone() 2024-08-09T14:56:00.549 INFO:tasks.ceph.osd.3.smithi155.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-08-09T14:56:00.549 INFO:tasks.ceph.osd.3.smithi155.stderr: 2024-08-09T14:56:00.983 INFO:tasks.ceph.osd.3.smithi155.stderr: -1141> 2024-08-09T14:55:59.933+0000 25a91640 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1345-g0a88632b/rpm/el9/BUILD/ceph-19.1.0-1345-g0a88632b/src/osd/PrimaryLogPG.cc: In function 'int PrimaryLogPG::start_cls_gather(PrimaryLogPG::OpContext*, std::map, ceph::buffer::v15_2_0::list>*, const string&, const char*, const char*, ceph::bufferlist&)' thread 25a91640 time 2024-08-09T14:55:59.762143+0000 2024-08-09T14:56:00.983 INFO:tasks.ceph.osd.3.smithi155.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1345-g0a88632b/rpm/el9/BUILD/ceph-19.1.0-1345-g0a88632b/src/osd/PrimaryLogPG.cc: 10387: FAILED ceph_assert(inserted) 2024-08-09T14:56:00.983 INFO:tasks.ceph.osd.3.smithi155.stderr: 2024-08-09T14:56:00.983 INFO:tasks.ceph.osd.3.smithi155.stderr: ceph version 19.1.0-1345-g0a88632b (0a88632b45f25ae38493667c7dbd155f6e59b499) squid (rc) 2024-08-09T14:56:00.984 INFO:tasks.ceph.osd.3.smithi155.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x5004c0] 2024-08-09T14:56:00.984 INFO:tasks.ceph.osd.3.smithi155.stderr: 2: ceph-osd(+0x3f867c) [0x50067c] 2024-08-09T14:56:00.984 INFO:tasks.ceph.osd.3.smithi155.stderr: 3: ceph-osd(+0x3976e0) [0x49f6e0] 2024-08-09T14:56:00.984 INFO:tasks.ceph.osd.3.smithi155.stderr: 4: (cls_cxx_gather(void*, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > > const&, std::__cxx11::basic_string, std::allocator > const&, char const*, char const*, ceph::buffer::v15_2_0::list&)+0x34a) [0x5f9fba] 2024-08-09T14:56:00.984 INFO:tasks.ceph.osd.3.smithi155.stderr: 5: /usr/lib64/rados-classes/libcls_test_remote_reads.so(+0x3861) [0x167c1861] 2024-08-09T14:56:00.984 INFO:tasks.ceph.osd.3.smithi155.stderr: 6: ceph-osd(+0x5e380d) [0x6eb80d] 2024-08-09T14:56:00.984 INFO:tasks.ceph.osd.3.smithi155.stderr: 7: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&)+0x5d) [0x6ec4bd] 2024-08-09T14:56:00.984 INFO:tasks.ceph.osd.3.smithi155.stderr: 8: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x594) [0x778ce4] 2024-08-09T14:56:00.984 INFO:tasks.ceph.osd.3.smithi155.stderr: 9: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x9e) [0x78f82e] 2024-08-09T14:56:00.984 INFO:tasks.ceph.osd.3.smithi155.stderr: 10: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x37e) [0x768b4e] 2024-08-09T14:56:00.984 INFO:tasks.ceph.osd.3.smithi155.stderr: 11: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x342b) [0x75fd3b] 2024-08-09T14:56:00.984 INFO:tasks.ceph.osd.3.smithi155.stderr: 12: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x69bed7] 2024-08-09T14:56:00.984 INFO:tasks.ceph.osd.3.smithi155.stderr: 13: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x8e6879] 2024-08-09T14:56:00.985 INFO:tasks.ceph.osd.3.smithi155.stderr: 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x6b6177] 2024-08-09T14:56:00.985 INFO:tasks.ceph.osd.3.smithi155.stderr: 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0xbb44ba] 2024-08-09T14:56:00.985 INFO:tasks.ceph.osd.3.smithi155.stderr: 16: ceph-osd(+0xaaca64) [0xbb4a64] 2024-08-09T14:56:00.985 INFO:tasks.ceph.osd.3.smithi155.stderr: 17: /lib64/libc.so.6(+0x89c02) [0x54dbc02] 2024-08-09T14:56:00.985 INFO:tasks.ceph.osd.3.smithi155.stderr: 18: clone() 2024-08-09T14:56:00.985 INFO:tasks.ceph.osd.3.smithi155.stderr: 2024-08-09T14:56:00.987 INFO:tasks.ceph.osd.3.smithi155.stderr: -1027> 2024-08-09T14:56:00.038+0000 25a91640 -1 *** Caught signal (Aborted) ** 2024-08-09T14:56:00.987 INFO:tasks.ceph.osd.3.smithi155.stderr: in thread 25a91640 thread_name:tp_osd_tp 2024-08-09T14:56:00.987 INFO:tasks.ceph.osd.3.smithi155.stderr: 2024-08-09T14:56:00.988 INFO:tasks.ceph.osd.3.smithi155.stderr: ceph version 19.1.0-1345-g0a88632b (0a88632b45f25ae38493667c7dbd155f6e59b499) squid (rc) 2024-08-09T14:56:00.988 INFO:tasks.ceph.osd.3.smithi155.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x54906f0] 2024-08-09T14:56:00.988 INFO:tasks.ceph.osd.3.smithi155.stderr: 2: /lib64/libc.so.6(+0x8b94c) [0x54dd94c] 2024-08-09T14:56:00.988 INFO:tasks.ceph.osd.3.smithi155.stderr: 3: raise() 2024-08-09T14:56:00.988 INFO:tasks.ceph.osd.3.smithi155.stderr: 4: abort() 2024-08-09T14:56:00.988 INFO:tasks.ceph.osd.3.smithi155.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x50051b] 2024-08-09T14:56:00.988 INFO:tasks.ceph.osd.3.smithi155.stderr: 6: ceph-osd(+0x3f867c) [0x50067c] 2024-08-09T14:56:00.988 INFO:tasks.ceph.osd.3.smithi155.stderr: 7: ceph-osd(+0x3976e0) [0x49f6e0] 2024-08-09T14:56:00.988 INFO:tasks.ceph.osd.3.smithi155.stderr: 8: (cls_cxx_gather(void*, std::set, std::allocator >, std::less, std::allocator > >, std::allocator, std::allocator > > > const&, std::__cxx11::basic_string, std::allocator > const&, char const*, char const*, ceph::buffer::v15_2_0::list&)+0x34a) [0x5f9fba] 2024-08-09T14:56:00.988 INFO:tasks.ceph.osd.3.smithi155.stderr: 9: /usr/lib64/rados-classes/libcls_test_remote_reads.so(+0x3861) [0x167c1861] 2024-08-09T14:56:00.988 INFO:tasks.ceph.osd.3.smithi155.stderr: 10: ceph-osd(+0x5e380d) [0x6eb80d] 2024-08-09T14:56:00.988 INFO:tasks.ceph.osd.3.smithi155.stderr: 11: (ClassHandler::ClassMethod::exec(void*, ceph::buffer::v15_2_0::list&, ceph::buffer::v15_2_0::list&)+0x5d) [0x6ec4bd] 2024-08-09T14:56:00.989 INFO:tasks.ceph.osd.3.smithi155.stderr: 12: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector >&)+0x594) [0x778ce4] 2024-08-09T14:56:00.989 INFO:tasks.ceph.osd.3.smithi155.stderr: 13: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x9e) [0x78f82e] 2024-08-09T14:56:00.989 INFO:tasks.ceph.osd.3.smithi155.stderr: 14: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x37e) [0x768b4e] 2024-08-09T14:56:00.989 INFO:tasks.ceph.osd.3.smithi155.stderr: 15: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x342b) [0x75fd3b] 2024-08-09T14:56:00.989 INFO:tasks.ceph.osd.3.smithi155.stderr: 16: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x197) [0x69bed7] 2024-08-09T14:56:00.989 INFO:tasks.ceph.osd.3.smithi155.stderr: 17: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x8e6879] 2024-08-09T14:56:00.989 INFO:tasks.ceph.osd.3.smithi155.stderr: 18: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xd07) [0x6b6177] 2024-08-09T14:56:00.989 INFO:tasks.ceph.osd.3.smithi155.stderr: 19: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x2aa) [0xbb44ba] 2024-08-09T14:56:00.989 INFO:tasks.ceph.osd.3.smithi155.stderr: 20: ceph-osd(+0xaaca64) [0xbb4a64] 2024-08-09T14:56:00.989 INFO:tasks.ceph.osd.3.smithi155.stderr: 21: /lib64/libc.so.6(+0x89c02) [0x54dbc02] 2024-08-09T14:56:00.989 INFO:tasks.ceph.osd.3.smithi155.stderr: 22: clone() 2024-08-09T14:56:00.989 INFO:tasks.ceph.osd.3.smithi155.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-08-09T14:56:00.989 INFO:tasks.ceph.osd.3.smithi155.stderr: 2024-08-09T14:56:01.253 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:01.182+0000 12a65640 -1 osd.0 233 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:01.447 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:01.385+0000 ea5d640 -1 osd.5 239 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4479.0:27 27.11 27:8e16cba4:::25.obj:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206) 2024-08-09T14:56:02.354 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:02.343+0000 12a65640 -1 osd.0 234 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:02.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:58.355+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:02.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:58.355+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:02.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:59.355+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:02.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:59.355+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:02.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:55:59.355+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:02.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:00.355+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:02.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:00.355+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:02.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:00.355+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:02.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:00.355+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:56:02.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:01.355+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:02.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:01.355+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:02.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:01.355+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:02.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.355+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:02.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.355+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:02.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.355+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:02.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.413+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:56:02.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.413+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:56:02.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.413+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:56:02.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.413+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:56:02.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.413+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:56:02.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.413+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=237}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:56:02.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.413+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:56:02.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.415+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 48 ==== osd_map(237..241 src has 1..241) ==== 3028+0+0 (secure 0 0 0) 0x7fc3bc036020 con 0x55a86d4642e0 2024-08-09T14:56:02.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.415+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(237..241 src has 1..241) 2024-08-09T14:56:02.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.415+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [237,241] > 236 2024-08-09T14:56:02.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.415+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 237 2024-08-09T14:56:02.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.415+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:02.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.415+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 237 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:02.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.415+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:02.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 238 2024-08-09T14:56:02.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:02.533 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:02.449+0000 ea5d640 -1 osd.5 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4479.0:27 27.11 27:8e16cba4:::25.obj:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206) 2024-08-09T14:56:03.510 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:03.439+0000 12a65640 -1 osd.0 236 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:03.692 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:03.684+0000 ea5d640 -1 osd.5 241 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4479.0:27 27.11 27:8e16cba4:::25.obj:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206) 2024-08-09T14:56:04.216 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:06:16.981 35989== 2024-08-09T14:56:04.216 INFO:tasks.ceph.osd.3.smithi155.stderr:==00:00:06:16.981 35989== Exit program on first error (--exit-on-first-error=yes) 2024-08-09T14:56:04.278 INFO:tasks.ceph.osd.3.smithi155.stderr:daemon-helper: command failed with exit status 42 2024-08-09T14:56:04.868 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:04.803+0000 12a65640 -1 osd.0 238 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:05.185 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:05.168+0000 ea5d640 -1 osd.5 242 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4479.0:27 27.11 27:8e16cba4:::25.obj:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206) 2024-08-09T14:56:05.356 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:40.599 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:05.359 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:40.603 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:05.438 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:06:40.681 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:05.451 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:40.695 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:05.455 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:40.699 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:05.460 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:40.707 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:05.483 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:40.730 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:05.553 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:06:40.797 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:05.578 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:06:40.824 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:06.070 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:05.930+0000 12a65640 -1 osd.0 238 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:06.406 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:06.383+0000 ea5d640 -1 osd.5 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4479.0:27 27.11 27:8e16cba4:::25.obj:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206) 2024-08-09T14:56:07.043 INFO:tasks.ceph.osd.6.smithi194.stderr:2024-08-09T14:56:07.024+0000 ea5d640 -1 osd.6 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4823.0:3 118.0 118.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e217) 2024-08-09T14:56:07.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 238 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:07.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:07.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 239 2024-08-09T14:56:07.415 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:07.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 239 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:07.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:07.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 240 2024-08-09T14:56:07.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:07.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 240 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:07.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:07.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 241 2024-08-09T14:56:07.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:07.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 241 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:07.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:07.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:56:07.416 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:02.416+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:56:07.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:03.355+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:07.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:03.355+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:07.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:03.355+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:07.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:04.356+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:07.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:04.356+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:07.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:04.356+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:07.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:05.356+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:07.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:05.356+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:07.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:05.356+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:07.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:06.356+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:07.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:06.356+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:07.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:06.356+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:07.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.356+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:07.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.356+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:07.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.356+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:07.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.414+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:56:07.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.414+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:56:07.530 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.414+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:56:07.530 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.414+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:56:07.530 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.414+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:56:07.531 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.414+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=242}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:56:07.531 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.414+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:56:07.531 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 49 ==== osd_map(242..246 src has 1..246) ==== 6634+0+0 (secure 0 0 0) 0x7fc3bc082070 con 0x55a86d4642e0 2024-08-09T14:56:07.531 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(242..246 src has 1..246) 2024-08-09T14:56:07.531 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [242,246] > 241 2024-08-09T14:56:07.531 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 242 2024-08-09T14:56:07.531 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:07.531 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 242 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:07.531 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:07.531 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 243 2024-08-09T14:56:07.531 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:07.531 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 243 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:07.531 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:07.532 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 244 2024-08-09T14:56:07.532 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:07.532 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 244 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:07.532 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:07.532 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 245 2024-08-09T14:56:07.532 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:07.532 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 245 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:07.532 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:07.532 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 246 2024-08-09T14:56:07.532 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:07.532 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 246 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:07.532 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:07.599 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:07.485+0000 12a65640 -1 osd.0 240 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:07.990 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:07.979+0000 ea5d640 -1 osd.5 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4479.0:27 27.11 27:8e16cba4:::25.obj:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206) 2024-08-09T14:56:09.329 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:09.321+0000 12a65640 -1 osd.0 242 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:09.441 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:09.406+0000 ea5d640 -1 osd.5 246 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4479.0:27 27.11 27:8e16cba4:::25.obj:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206) 2024-08-09T14:56:10.491 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:10.486+0000 12a65640 -1 osd.0 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:10.705 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:10.547+0000 ea5d640 -1 osd.5 246 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4479.0:27 27.11 27:8e16cba4:::25.obj:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206) 2024-08-09T14:56:11.013 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-08-09T14:56:11.632 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:11.556+0000 ea5d640 -1 osd.5 247 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4479.0:27 27.11 27:8e16cba4:::25.obj:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206) 2024-08-09T14:56:11.849 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:11.839+0000 12a65640 -1 osd.0 244 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:12.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:56:12.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:07.528+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:56:12.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:08.356+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:12.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:08.356+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:12.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:08.356+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:12.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:09.356+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:12.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:09.356+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:12.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:09.356+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:12.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:10.356+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:12.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:10.356+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:12.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:10.356+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:12.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:10.356+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:56:12.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:11.356+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:12.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:11.356+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:12.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:11.356+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:12.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.356+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:12.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.356+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:12.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.356+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:12.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.415+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:56:12.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.415+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:56:12.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.415+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:56:12.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.415+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:56:12.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.415+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:56:12.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.415+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=247}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:56:12.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.415+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:56:12.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 50 ==== osd_map(247..250 src has 1..250) ==== 2134+0+0 (secure 0 0 0) 0x7fc3bc036060 con 0x55a86d4642e0 2024-08-09T14:56:12.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(247..250 src has 1..250) 2024-08-09T14:56:12.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [247,250] > 246 2024-08-09T14:56:12.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 247 2024-08-09T14:56:12.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:12.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 247 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:12.422 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:12.786 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:12.780+0000 ea5d640 -1 osd.5 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4479.0:27 27.11 27:8e16cba4:::25.obj:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206) 2024-08-09T14:56:13.805 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:13.787+0000 ea5d640 -1 osd.5 249 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4479.0:27 27.11 27:8e16cba4:::25.obj:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206) 2024-08-09T14:56:14.223 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:14.113+0000 12a65640 -1 osd.0 247 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:15.418 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:15.411+0000 12a65640 -1 osd.0 248 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:15.604 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:15.526+0000 ea5d640 -1 osd.5 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4479.0:27 27.11 27:8e16cba4:::25.obj:head [write 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206) 2024-08-09T14:56:16.610 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:16.591+0000 12a65640 -1 osd.0 250 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:17.223 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~6s 2024-08-09T14:56:17.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 248 2024-08-09T14:56:17.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:17.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 248 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:17.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:17.417 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 249 2024-08-09T14:56:17.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:17.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 249 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:17.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:17.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 250 2024-08-09T14:56:17.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:17.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 250 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:17.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:17.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:56:17.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:12.418+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:56:17.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:13.356+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:17.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:13.356+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:17.418 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:13.356+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:17.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:14.356+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:17.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:14.356+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:17.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:14.356+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:17.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:15.356+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:17.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:15.356+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:17.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:15.357+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:17.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:16.357+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:17.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:16.357+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:17.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:16.357+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:17.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.357+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:17.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.357+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:17.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.357+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:17.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.416+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:56:17.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.416+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:56:17.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.416+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:56:17.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.416+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:56:17.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.416+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:56:17.771 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:17.760+0000 12a65640 -1 osd.0 251 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:18.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.416+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=251}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:56:18.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.416+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:56:18.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 51 ==== osd_map(251..255 src has 1..255) ==== 1352+0+0 (secure 0 0 0) 0x7fc3bc082020 con 0x55a86d4642e0 2024-08-09T14:56:18.358 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(251..255 src has 1..255) 2024-08-09T14:56:18.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [251,255] > 250 2024-08-09T14:56:18.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 251 2024-08-09T14:56:18.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:18.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 251 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:18.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:18.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 252 2024-08-09T14:56:18.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:18.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 252 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:18.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:18.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 253 2024-08-09T14:56:18.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:18.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 253 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:18.359 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:18.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 254 2024-08-09T14:56:18.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:18.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 254 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:18.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:18.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 255 2024-08-09T14:56:18.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:18.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 255 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:18.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:18.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:56:18.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:17.419+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:56:19.008 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:18.905+0000 12a65640 -1 osd.0 252 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:20.209 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:20.203+0000 12a65640 -1 osd.0 253 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:21.562 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:21.388+0000 12a65640 -1 osd.0 256 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:22.422 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:18.357+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:22.422 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:18.357+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:22.422 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:18.357+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:22.422 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:19.357+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:22.423 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:19.357+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:22.423 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:19.357+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:22.423 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:20.357+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:22.423 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:20.357+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:22.423 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:20.357+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:22.423 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:20.357+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:56:22.423 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:21.357+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:22.423 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:21.357+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:22.423 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:21.357+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:22.423 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.357+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:22.423 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.357+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:22.423 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.357+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:22.424 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.417+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:56:22.424 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.417+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:56:22.424 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.417+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:56:22.424 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.417+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:56:22.424 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.417+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:56:22.424 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.417+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=256}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:56:22.424 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.417+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:56:22.424 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 52 ==== osd_map(256..260 src has 1..260) ==== 4347+0+0 (secure 0 0 0) 0x7fc3bc07e020 con 0x55a86d4642e0 2024-08-09T14:56:22.424 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(256..260 src has 1..260) 2024-08-09T14:56:22.424 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [256,260] > 255 2024-08-09T14:56:22.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 256 2024-08-09T14:56:22.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:22.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 256 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:22.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:22.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 257 2024-08-09T14:56:22.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:23.372 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:23.311+0000 12a65640 -1 osd.0 257 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4700.0:7 80.18 80:1c3bf51a:test-rados-api-smithi155-41609-4::ws:head [write-same in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-08-09T14:56:23.432 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~12s 2024-08-09T14:56:24.363 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:24.355+0000 12a65640 -1 osd.0 257 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4700.0:7 80.18 80:1c3bf51a:test-rados-api-smithi155-41609-4::ws:head [write-same in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e159) 2024-08-09T14:56:25.561 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:00.804 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:25.571 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:00.817 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:25.574 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:00.821 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:25.578 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:00.821 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:25.583 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:00.827 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:25.586 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:00.829 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:25.736 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:00.979 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:25.780 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:01.023 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:25.785 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:01.031 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:26.332 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:26.264+0000 12a65640 -1 osd.0 260 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:27.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 257 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:27.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:27.419 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 258 2024-08-09T14:56:27.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:27.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 258 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:27.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:27.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 259 2024-08-09T14:56:27.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:27.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 259 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:27.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:27.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 260 2024-08-09T14:56:27.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:27.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 260 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:27.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:27.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:56:27.420 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:22.421+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:56:27.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:23.357+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:27.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:23.357+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:27.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:23.357+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:27.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:24.357+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:27.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:24.357+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:27.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:24.357+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:27.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:25.357+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:27.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:25.357+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:27.421 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:25.357+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:27.422 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:26.357+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:27.422 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:26.357+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:27.422 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:26.357+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:27.422 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.358+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:27.422 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.358+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:27.422 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.358+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:27.423 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.418+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:56:27.423 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.418+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:56:27.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.418+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:56:27.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.418+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:56:27.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.418+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:56:27.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.418+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=261}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:56:27.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.418+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:56:27.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 53 ==== osd_map(261..265 src has 1..265) ==== 4534+0+0 (secure 0 0 0) 0x7fc3bc082070 con 0x55a86d4642e0 2024-08-09T14:56:27.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(261..265 src has 1..265) 2024-08-09T14:56:27.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [261,265] > 260 2024-08-09T14:56:27.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 261 2024-08-09T14:56:27.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:27.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 261 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:27.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:27.435 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 262 2024-08-09T14:56:27.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:27.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 262 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:27.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:27.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 263 2024-08-09T14:56:27.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:27.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 263 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:27.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:27.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 264 2024-08-09T14:56:27.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:27.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 264 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:27.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:27.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 265 2024-08-09T14:56:27.436 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:27.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 265 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:27.437 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:27.670 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:27.562+0000 12a65640 -1 osd.0 260 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:28.807 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:28.792+0000 12a65640 -1 osd.0 261 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:29.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~19s 2024-08-09T14:56:29.985 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:29.882+0000 12a65640 -1 osd.0 262 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:31.683 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:31.551+0000 12a65640 -1 osd.0 264 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4861.0:3 96.13 96:cd8c8e4c:::foo2:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e194) 2024-08-09T14:56:32.424 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:56:32.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:27.433+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:56:32.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:28.358+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:32.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:28.358+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:32.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:28.358+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:32.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:29.358+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:32.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:29.358+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:32.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:29.358+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:32.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:30.358+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:32.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:30.358+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:32.425 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:30.358+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:32.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:30.358+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:56:32.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:31.358+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:32.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:31.358+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:32.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:31.358+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:32.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.358+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:32.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.358+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:32.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.358+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:32.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.419+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:56:32.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.419+0000 7fc3cb426640 2 client.4435.objecter tid 18 on osd.0 is laggy 2024-08-09T14:56:32.426 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.419+0000 7fc3cb426640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-08-09T14:56:32.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.419+0000 7fc3cb426640 10 monclient: _renew_subs 2024-08-09T14:56:32.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.419+0000 7fc3cb426640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.194:3300/0 2024-08-09T14:56:32.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.419+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=266}) -- 0x7fc39c004830 con 0x55a86d4642e0 2024-08-09T14:56:32.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.419+0000 7fc3cb426640 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- ping magic: 0 -- 0x7fc39c004ac0 con 0x55a86d4e4550 2024-08-09T14:56:32.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== mon.1 v2:172.21.15.194:3300/0 54 ==== osd_map(266..270 src has 1..270) ==== 3161+0+0 (secure 0 0 0) 0x7fc3bc07e070 con 0x55a86d4642e0 2024-08-09T14:56:32.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(266..270 src has 1..270) 2024-08-09T14:56:32.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [266,270] > 265 2024-08-09T14:56:32.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 266 2024-08-09T14:56:32.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:32.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 266 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:32.427 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:34.169 INFO:tasks.workunit.client.0.smithi155.stdout: api_asi ec_io: Running main() from gmock_main.cc 2024-08-09T14:56:34.169 INFO:tasks.workunit.client.0.smithi155.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-08-09T14:56:34.169 INFO:tasks.workunit.client.0.smithi155.stdout: ec_io: [----------] Global test environment set-up. 2024-08-09T14:56:34.169 INFO:tasks.workunit.client.0.smithi155.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-08-09T14:56:34.169 INFO:tasks.workunit.client.0.smithi155.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-08-09T14:56:34.170 INFO:tasks.workunit.client.0.smithi155.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (9281 ms) 2024-08-09T14:56:34.170 INFO:tasks.workunit.client.0.smithi155.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-08-09T14:56:34.170 INFO:tasks.workunit.client.0.smithi155.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (277235 ms) 2024-08-09T14:56:34.170 INFO:tasks.workunit.client.0.smithi155.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (286516 ms total) 2024-08-09T14:56:34.170 INFO:tasks.workunit.client.0.smithi155.stdout: ec_io: 2024-08-09T14:56:34.170 INFO:tasks.workunit.client.0.smithi155.stdout: ec_io: [----------] Global test environment tear-down 2024-08-09T14:56:34.170 INFO:tasks.workunit.client.0.smithi155.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (286516 ms total) 2024-08-09T14:56:34.170 INFO:tasks.workunit.client.0.smithi155.stdout: ec_io: [ PASSED ] 2 tests. 2024-08-09T14:56:34.745 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:34.623+0000 12a65640 -1 osd.0 268 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:31 25.d 25:b518f988:::29.obj:head [write 0~29 in=29b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224) 2024-08-09T14:56:35.772 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:35.748+0000 12a65640 -1 osd.0 268 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:31 25.d 25:b518f988:::29.obj:head [write 0~29 in=29b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224) 2024-08-09T14:56:35.852 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~25s 2024-08-09T14:56:37.016 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:37.009+0000 12a65640 -1 osd.0 269 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:31 25.d 25:b518f988:::29.obj:head [write 0~29 in=29b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224) 2024-08-09T14:56:37.139 INFO:tasks.workunit.client.0.smithi155.stdout:o: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 267 2024-08-09T14:56:37.139 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:37.139 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 267 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:37.140 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:37.140 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 268 2024-08-09T14:56:37.140 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:37.140 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 268 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:37.140 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:37.140 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 269 2024-08-09T14:56:37.140 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:37.140 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 269 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:37.140 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:37.140 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 270 2024-08-09T14:56:37.140 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 10 client.4435.objecter checking op 18 2024-08-09T14:56:37.140 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 270 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:37.140 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:37.141 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:56:37.141 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:32.423+0000 7fc3ba7fc640 20 client.4435.objecter 18 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:56:37.141 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:33.358+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:37.141 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:33.358+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:37.141 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:33.358+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:37.141 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:34.358+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:37.141 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:34.358+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:37.141 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:34.358+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:37.141 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:35.358+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:37.141 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:35.358+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:37.141 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:35.358+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:37.141 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:36.358+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:37.141 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:36.358+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:37.142 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:36.358+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:37.142 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 23 ==== osd_op_reply(18 exist [writefull 0~5] v162'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:56:38.316 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:38.153+0000 12a65640 -1 osd.0 270 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:31 25.d 25:b518f988:::29.obj:head [write 0~29 in=29b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224) 2024-08-09T14:56:39.322 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:39.238+0000 12a65640 -1 osd.0 271 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:31 25.d 25:b518f988:::29.obj:head [write 0~29 in=29b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224) 2024-08-09T14:56:39.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(18 exist [writefull 0~5] v162'5 uv5 ondisk = 0) 2024-08-09T14:56:39.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:56:39.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 18 ondisk uv 5 in 20.1 attempt 0 2024-08-09T14:56:39.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 0 2024-08-09T14:56:39.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 18 2024-08-09T14:56:39.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 18 2024-08-09T14:56:39.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:56:39.360 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 18 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3c99c3640 5 client.4435.objecter 0 in flight 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3c99c3640 20 librados: complete_aio_write 0x55a86d4e3d00 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (119300 ms) 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3cb6c6780 20 librados: queue_aio_write 0x55a86d4f5580 completion 0x55a86d4e3e10 write_seq 5 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d431b00 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 270 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid exist '@20' '@20' [writefull 0~5 in=5b] tid 19 osd.0 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 19 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 19 to 20.1 on osd.0 2024-08-09T14:56:39.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:19 20.1 20:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270) -- 0x55a86d5a9b70 con 0x55a86d4e4550 2024-08-09T14:56:39.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:56:39.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.138+0000 7fc3cb6c6780 5 client.4435.objecter 1 in flight 2024-08-09T14:56:39.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.358+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:39.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.358+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:39.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.358+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:39.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:37.420+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:56:39.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:38.359+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:39.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:38.359+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:39.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:38.359+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:39.876 INFO:tasks.workunit.client.0.smithi155.stdout: cls: Running main() from gmock_main.cc 2024-08-09T14:56:39.876 INFO:tasks.workunit.client.0.smithi155.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-08-09T14:56:39.876 INFO:tasks.workunit.client.0.smithi155.stdout: cls: [----------] Global test environment set-up. 2024-08-09T14:56:39.876 INFO:tasks.workunit.client.0.smithi155.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-08-09T14:56:39.876 INFO:tasks.workunit.client.0.smithi155.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-08-09T14:56:39.876 INFO:tasks.workunit.client.0.smithi155.stdout: cls: [ OK ] NeoRadosCls.DNE (81173 ms) 2024-08-09T14:56:39.877 INFO:tasks.workunit.client.0.smithi155.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-08-09T14:56:39.877 INFO:tasks.workunit.client.0.smithi155.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (211112 ms) 2024-08-09T14:56:39.877 INFO:tasks.workunit.client.0.smithi155.stdout: cls: [----------] 2 tests from NeoRadosCls (292285 ms total) 2024-08-09T14:56:39.877 INFO:tasks.workunit.client.0.smithi155.stdout: cls: 2024-08-09T14:56:39.877 INFO:tasks.workunit.client.0.smithi155.stdout: cls: [----------] Global test environment tear-down 2024-08-09T14:56:39.877 INFO:tasks.workunit.client.0.smithi155.stdout: cls: [==========] 2 tests from 1 test suite ran. (292286 ms total) 2024-08-09T14:56:39.877 INFO:tasks.workunit.client.0.smithi155.stdout: cls: [ PASSED ] 2 tests. 2024-08-09T14:56:40.834 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:40.825+0000 12a65640 -1 osd.0 272 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:31 25.d 25:b518f988:::29.obj:head [write 0~29 in=29b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224) 2024-08-09T14:56:42.062 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~31s 2024-08-09T14:56:42.327 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:42.140+0000 12a65640 -1 osd.0 273 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4513.0:31 25.d 25:b518f988:::29.obj:head [write 0~29 in=29b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e224) 2024-08-09T14:56:45.724 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:20.967 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:45.744 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:20.987 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:45.764 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:21.010 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:45.765 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:21.008 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:45.880 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:21.123 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:45.912 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:21.159 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:45.990 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:21.233 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:46.004 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:21.247 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:46.023 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:21.270 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:46.923 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:39.359+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:46.924 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:39.359+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:46.924 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:39.359+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:46.924 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:40.359+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:46.924 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:40.359+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:46.924 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:40.359+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:46.924 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:40.359+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:56:46.924 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:41.359+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:46.924 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:41.359+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:46.924 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:41.359+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:46.924 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:42.359+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:46.925 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:42.359+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:46.925 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:42.359+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:46.925 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:42.421+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:56:46.925 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:43.360+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:46.925 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:43.360+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:46.925 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:43.360+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:46.925 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:44.359+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:46.925 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:44.359+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:46.925 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:44.359+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:46.925 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:45.359+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:46.925 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:45.359+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:46.925 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:45.359+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:46.925 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.359+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:46.926 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.359+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:46.926 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.359+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:46.926 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 24 ==== osd_op_reply(19 exist [writefull 0~5] v270'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:56:46.926 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(19 exist [writefull 0~5] v270'6 uv6 ondisk = 0) 2024-08-09T14:56:46.926 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:56:46.926 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 19 ondisk uv 6 in 20.1 attempt 0 2024-08-09T14:56:46.926 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 0 2024-08-09T14:56:46.926 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 19 2024-08-09T14:56:46.926 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 19 2024-08-09T14:56:46.926 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:56:47.228 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:47.120+0000 ea5d640 -1 osd.5 278 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4760.0:5 100.5s0 100:a828f945:test-rados-api-smithi155-41823-10::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228) 2024-08-09T14:56:48.194 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:48.178+0000 ea5d640 -1 osd.5 279 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4760.0:5 100.5s0 100:a828f945:test-rados-api-smithi155-41823-10::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228) 2024-08-09T14:56:48.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~37s 2024-08-09T14:56:48.798 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 19 2024-08-09T14:56:48.798 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3c99c3640 5 client.4435.objecter 0 in flight 2024-08-09T14:56:48.798 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3c99c3640 20 librados: complete_aio_write 0x55a86d4e3e10 2024-08-09T14:56:48.798 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (9784 ms) 2024-08-09T14:56:48.798 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-08-09T14:56:48.798 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3cb6c6780 20 librados: queue_aio_write 0x55a86d4f5580 completion 0x55a86d432280 write_seq 6 2024-08-09T14:56:48.798 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit op 0x55a86d5a92e0 2024-08-09T14:56:48.799 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target epoch 270 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:48.799 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3cb6c6780 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:48.799 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3cb6c6780 10 client.4435.objecter _calc_target raw pgid 20.d3a59201 -> actual 20.1 acting [0,5] primary 0 2024-08-09T14:56:48.799 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3cb6c6780 20 client.4435.objecter _get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:56:48.799 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3cb6c6780 10 client.4435.objecter _op_submit oid exist '@20' '@20' [writefull 0~5 in=5b] tid 20 osd.0 2024-08-09T14:56:48.799 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3cb6c6780 20 client.4435.objecter get_session s=0x55a86d4f2940 osd=0 3 2024-08-09T14:56:48.799 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3cb6c6780 15 client.4435.objecter _session_op_assign 0 20 2024-08-09T14:56:48.799 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3cb6c6780 15 client.4435.objecter _send_op 20 to 20.1 on osd.0 2024-08-09T14:56:48.799 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 --> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] -- osd_op(unknown.0.0:20 20.1 20:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e270) -- 0x55a86d5a9790 con 0x55a86d4e4550 2024-08-09T14:56:48.799 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:56:48.799 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:46.922+0000 7fc3cb6c6780 5 client.4435.objecter 1 in flight 2024-08-09T14:56:48.799 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:47.359+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:48.799 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:47.359+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:48.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:47.359+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:48.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:47.421+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:56:48.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.360+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:48.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.360+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:48.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.360+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:48.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 25 ==== osd_map(271..277 src has 1..281) ==== 4967+0+0 (crc 0 0 0) 0x7fc3c00085b0 con 0x55a86d4e4550 2024-08-09T14:56:48.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_map(271..277 src has 1..281) 2024-08-09T14:56:48.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map got epochs [271,277] > 270 2024-08-09T14:56:48.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 271 2024-08-09T14:56:48.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 10 client.4435.objecter checking op 20 2024-08-09T14:56:48.800 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 271 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:49.831 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:49.613+0000 ea5d640 -1 osd.5 281 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4760.0:5 100.5s0 100:a828f945:test-rados-api-smithi155-41823-10::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228) 2024-08-09T14:56:50.287 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:25.530 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:50.290 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:25.534 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:50.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:50.361 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 272 2024-08-09T14:56:50.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 10 client.4435.objecter checking op 20 2024-08-09T14:56:50.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 272 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:50.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:50.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 273 2024-08-09T14:56:50.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 10 client.4435.objecter checking op 20 2024-08-09T14:56:50.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 273 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:50.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:50.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 274 2024-08-09T14:56:50.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 10 client.4435.objecter checking op 20 2024-08-09T14:56:50.362 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 274 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:50.363 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:50.363 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 275 2024-08-09T14:56:50.363 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 10 client.4435.objecter checking op 20 2024-08-09T14:56:50.363 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 275 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:50.363 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:50.363 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 276 2024-08-09T14:56:50.363 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 10 client.4435.objecter checking op 20 2024-08-09T14:56:50.363 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 276 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:50.363 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:50.363 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 3 client.4435.objecter handle_osd_map decoding incremental epoch 277 2024-08-09T14:56:50.363 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 10 client.4435.objecter checking op 20 2024-08-09T14:56:50.363 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target epoch 277 base exist @20 precalc_pgid 0 pgid 0.0 is_write 2024-08-09T14:56:50.364 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter _calc_target target exist @20 -> pgid 20.d3a59201 2024-08-09T14:56:50.364 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter dump_active .. 0 homeless 2024-08-09T14:56:50.364 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:48.796+0000 7fc3ba7fc640 20 client.4435.objecter 20 20.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-08-09T14:56:50.364 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:49.360+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:50.364 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:49.360+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:50.364 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:49.360+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:50.433 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:07:25.676 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:50.649 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:50.640+0000 ea5d640 -1 osd.5 281 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4760.0:5 100.5s0 100:a828f945:test-rados-api-smithi155-41823-10::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228) 2024-08-09T14:56:50.896 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:26.139 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:50.971 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:26.214 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:51.149 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:07:26.392 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:51.203 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:26.449 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:51.214 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:26.460 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:51.318 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:07:26.565 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:56:52.027 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:51.922+0000 ea5d640 -1 osd.5 283 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4760.0:5 100.5s0 100:a828f945:test-rados-api-smithi155-41823-10::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228) 2024-08-09T14:56:53.393 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:53.322+0000 ea5d640 -1 osd.5 285 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4760.0:5 100.5s0 100:a828f945:test-rados-api-smithi155-41823-10::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228) 2024-08-09T14:56:54.462 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:54.353+0000 ea5d640 -1 osd.5 285 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4760.0:5 100.5s0 100:a828f945:test-rados-api-smithi155-41823-10::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228) 2024-08-09T14:56:54.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~43s 2024-08-09T14:56:55.974 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:55.896+0000 ea5d640 -1 osd.5 286 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4760.0:5 100.5s0 100:a828f945:test-rados-api-smithi155-41823-10::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e228) 2024-08-09T14:56:56.891 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:56.886+0000 ea5d640 -1 osd.5 286 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4936.0:7 114.1s0 114:9357852e:test-rados-api-smithi155-41617-14::foo:head [call lock.lock in=62b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247) 2024-08-09T14:56:57.241 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:50.360+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:57.241 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:50.360+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:57.241 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:50.360+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:57.241 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:50.360+0000 7fc3b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-08-09T14:56:57.241 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:51.360+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:57.241 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:51.360+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:57.242 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:51.360+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:57.242 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:52.361+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:57.242 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:52.361+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:57.242 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:52.361+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:57.242 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:52.422+0000 7fc3cb426640 10 client.4435.objecter tick 2024-08-09T14:56:57.242 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:53.361+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:57.242 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:53.361+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:57.242 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:53.361+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:57.242 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:54.361+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:57.242 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:54.361+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:57.242 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:54.361+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:57.242 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:55.361+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:57.243 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:55.361+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:57.243 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:55.361+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:57.243 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:56.361+0000 7fc3b97fa640 10 monclient: tick 2024-08-09T14:56:57.243 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:56.361+0000 7fc3b97fa640 10 monclient: _check_auth_tickets 2024-08-09T14:56:57.243 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:56.361+0000 7fc3b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-08-09T14:56:57.243 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.228+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 <== osd.0 v2:172.21.15.155:6820/729861593 26 ==== osd_op_reply(20 exist [writefull 0~5] v277'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7fc3c0008a30 con 0x55a86d4e4550 2024-08-09T14:56:57.243 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.228+0000 7fc3c99c3640 10 client.4435.objecter ms_dispatch 0x55a86d3d3e10 osd_op_reply(20 exist [writefull 0~5] v277'7 uv7 ondisk = 0) 2024-08-09T14:56:57.243 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.228+0000 7fc3c99c3640 10 client.4435.objecter in handle_osd_op_reply 2024-08-09T14:56:57.243 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.228+0000 7fc3c99c3640 7 client.4435.objecter handle_osd_op_reply 20 ondisk uv 7 in 20.1 attempt 0 2024-08-09T14:56:57.243 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.228+0000 7fc3c99c3640 10 client.4435.objecter op 0 rval 0 len 0 2024-08-09T14:56:57.243 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.228+0000 7fc3c99c3640 15 client.4435.objecter handle_osd_op_reply completed tid 20 2024-08-09T14:56:57.243 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.228+0000 7fc3c99c3640 15 client.4435.objecter _finish_op 20 2024-08-09T14:56:57.243 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.228+0000 7fc3c99c3640 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 4 2024-08-09T14:56:57.244 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.228+0000 7fc3c99c3640 15 client.4435.objecter _session_op_remove 0 20 2024-08-09T14:56:57.244 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.228+0000 7fc3c99c3640 5 client.4435.objecter 0 in flight 2024-08-09T14:56:57.244 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.228+0000 7fc3c99c3640 20 librados: complete_aio_write 0x55a86d432280 2024-08-09T14:56:57.244 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (10317 ms) 2024-08-09T14:56:57.248 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.239+0000 7fc3cb6c6780 10 librados: watch_flush enter 2024-08-09T14:56:57.248 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 10 librados: watch_flush exit 2024-08-09T14:56:57.248 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 10 client.4435.objecter close_session for osd.0 2024-08-09T14:56:57.248 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 >> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] conn(0x55a86d4e4550 msgr2=0x55a86d5a8a00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:56:57.248 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 1 --2- 172.21.15.155:0/3100773278 >> [v2:172.21.15.155:6820/729861593,v1:172.21.15.155:6822/729861593] conn(0x55a86d4e4550 0x55a86d5a8a00 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T14:56:57.248 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 20 client.4435.objecter put_session s=0x55a86d4f2940 osd=0 1 2024-08-09T14:56:57.249 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 20 client.4435.objecter shutdown clearing up homeless session... 2024-08-09T14:56:57.249 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 10 client.4435.objecter successfully canceled tick 2024-08-09T14:56:57.249 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fc3ac06ba30 msgr2=0x7fc3ac06def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:56:57.249 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 1 --2- 172.21.15.155:0/3100773278 >> [v2:172.21.15.155:6800/689799774,v1:172.21.15.155:6801/689799774] conn(0x7fc3ac06ba30 0x7fc3ac06def0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4055000 tx=0x7fc3c404f000 comp rx=0 tx=0).stop 2024-08-09T14:56:57.249 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 10 monclient: shutdown 2024-08-09T14:56:57.249 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 20 monclient: shutdown discarding 0 pending message(s) 2024-08-09T14:56:57.249 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a86d4642e0 msgr2=0x55a86d4f6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-08-09T14:56:57.249 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 1 --2- 172.21.15.155:0/3100773278 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a86d4642e0 0x55a86d4f6b00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc00b6b0 tx=0x7fc3bc000ed0 comp rx=0 tx=0).stop 2024-08-09T14:56:57.249 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3c99c3640 1 -- 172.21.15.155:0/3100773278 reap_dead start 2024-08-09T14:56:57.249 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 shutdown_connections 2024-08-09T14:56:57.249 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 >> 172.21.15.155:0/3100773278 conn(0x55a86d46a2b0 msgr2=0x55a86d42af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-08-09T14:56:57.249 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 shutdown_connections 2024-08-09T14:56:57.249 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 1 -- 172.21.15.155:0/3100773278 wait complete. 2024-08-09T14:56:57.250 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.240+0000 7fc3cb6c6780 1 librados: shutdown 2024-08-09T14:56:57.250 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [----------] 12 tests from AsioRados (229214 ms total) 2024-08-09T14:56:57.250 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: 2024-08-09T14:56:57.250 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [----------] Global test environment tear-down 2024-08-09T14:56:57.250 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (309980 ms total) 2024-08-09T14:56:57.250 INFO:tasks.workunit.client.0.smithi155.stdout: api_asio: [ PASSED ] 12 tests. 2024-08-09T14:56:57.794 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:57.686+0000 12a65640 -1 osd.0 285 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:55:32.133805+0000 front 2024-08-09T14:56:56.422539+0000 (oldest deadline 2024-08-09T14:56:55.146494+0000) 2024-08-09T14:56:58.042 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T14:56:57.987+0000 ea5d640 -1 osd.5 287 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4936.0:7 114.1s0 114:9357852e:test-rados-api-smithi155-41617-14::foo:head [call lock.lock in=62b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e247) 2024-08-09T14:56:58.977 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T14:56:58.822+0000 12a65640 -1 osd.0 286 heartbeat_check: no reply from 172.21.15.194:6812 osd.6 since back 2024-08-09T14:55:32.133805+0000 front 2024-08-09T14:56:58.086190+0000 (oldest deadline 2024-08-09T14:56:55.146494+0000) 2024-08-09T14:57:00.691 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~50s 2024-08-09T14:57:06.899 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~56s 2024-08-09T14:57:08.080 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: Running main() from gmock_main.cc 2024-08-09T14:57:08.080 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-08-09T14:57:08.080 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [----------] Global test environment set-up. 2024-08-09T14:57:08.081 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-08-09T14:57:08.081 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: seed 41515 2024-08-09T14:57:08.081 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-08-09T14:57:08.081 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-08-09T14:57:08.081 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-08-09T14:57:08.081 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (47633 ms) 2024-08-09T14:57:08.081 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-08-09T14:57:08.081 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (8282 ms) 2024-08-09T14:57:08.081 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-08-09T14:57:08.081 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (19670 ms) 2024-08-09T14:57:08.081 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-08-09T14:57:08.081 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (5767 ms) 2024-08-09T14:57:08.081 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-08-09T14:57:08.081 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (9969 ms) 2024-08-09T14:57:08.082 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-08-09T14:57:08.082 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (14315 ms) 2024-08-09T14:57:08.082 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-08-09T14:57:08.082 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2713 ms) 2024-08-09T14:57:08.082 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-08-09T14:57:08.082 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (3660 ms) 2024-08-09T14:57:08.082 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-08-09T14:57:08.082 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (7740 ms) 2024-08-09T14:57:08.082 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-08-09T14:57:08.082 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (2392 ms) 2024-08-09T14:57:08.082 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-08-09T14:57:08.082 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (2379 ms) 2024-08-09T14:57:08.083 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-08-09T14:57:08.083 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (507 ms) 2024-08-09T14:57:08.083 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-08-09T14:57:08.083 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (5535 ms) 2024-08-09T14:57:08.083 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-08-09T14:57:08.083 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (3582 ms) 2024-08-09T14:57:08.083 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-08-09T14:57:08.083 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (53178 ms) 2024-08-09T14:57:08.083 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-08-09T14:57:08.083 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (13386 ms) 2024-08-09T14:57:08.083 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-08-09T14:57:08.083 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (38976 ms) 2024-08-09T14:57:08.083 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-08-09T14:57:08.084 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (19935 ms) 2024-08-09T14:57:08.084 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-08-09T14:57:08.084 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (7785 ms) 2024-08-09T14:57:08.084 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (267404 ms total) 2024-08-09T14:57:08.084 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: 2024-08-09T14:57:08.084 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-08-09T14:57:08.084 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-08-09T14:57:08.084 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (20640 ms) 2024-08-09T14:57:08.084 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-08-09T14:57:08.084 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (24779 ms) 2024-08-09T14:57:08.084 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-08-09T14:57:08.084 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (349 ms) 2024-08-09T14:57:08.084 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-08-09T14:57:08.085 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (171 ms) 2024-08-09T14:57:08.085 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-08-09T14:57:13.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~62s 2024-08-09T14:57:17.075 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-08-09T14:57:17.053+0000 ffc8640 -1 mon.c@2(peon).paxos(paxos updating c 252..764) lease_expire from mon.0 v2:172.21.15.155:3300/0 is 1.624788880s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2024-08-09T14:57:19.320 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~68s 2024-08-09T14:57:20.625 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibR api_list: [==========] Running 11 tests from 3 test suites. 2024-08-09T14:57:20.625 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: [----------] Global test environment set-up. 2024-08-09T14:57:20.626 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: [----------] 7 tests from LibRadosList 2024-08-09T14:57:20.626 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-08-09T14:57:20.626 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: [ OK ] LibRadosList.ListObjects (163525 ms) 2024-08-09T14:57:20.626 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-08-09T14:57:20.626 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (146253 ms) 2024-08-09T14:57:20.626 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-08-09T14:57:20.626 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: myset foo1,foo2,foo3 2024-08-09T14:57:20.626 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: foo1 2024-08-09T14:57:20.626 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: foo2 2024-08-09T14:57:20.626 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: foo3 2024-08-09T14:57:20.626 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: myset foo1,foo4,foo5 2024-08-09T14:57:20.627 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: foo4 2024-08-09T14:57:20.627 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: foo5 2024-08-09T14:57:20.627 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: foo1 2024-08-09T14:57:20.627 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: myset foo6,foo7 2024-08-09T14:57:20.627 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: foo7 2024-08-09T14:57:20.627 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: foo6 2024-08-09T14:57:20.627 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-08-09T14:57:20.627 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: ns1:foo4 2024-08-09T14:57:20.627 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: ns1:foo5 2024-08-09T14:57:20.627 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: ns2:foo7 2024-08-09T14:57:20.627 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: ns2:foo6 2024-08-09T14:57:20.627 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: ns1:foo1 2024-08-09T14:57:20.627 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: :foo1 2024-08-09T14:57:20.628 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: :foo2 2024-08-09T14:57:20.628 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: :foo3 2024-08-09T14:57:20.628 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (12316 ms) 2024-08-09T14:57:20.628 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-08-09T14:57:20.628 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 1 0 2024-08-09T14:57:20.628 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 10 0 2024-08-09T14:57:20.628 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 13 0 2024-08-09T14:57:20.628 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 7 0 2024-08-09T14:57:20.628 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 14 0 2024-08-09T14:57:20.628 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 0 0 2024-08-09T14:57:20.628 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 15 0 2024-08-09T14:57:20.628 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 11 0 2024-08-09T14:57:20.628 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 5 0 2024-08-09T14:57:20.628 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 8 0 2024-08-09T14:57:20.629 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 6 0 2024-08-09T14:57:20.629 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 3 0 2024-08-09T14:57:20.629 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 4 0 2024-08-09T14:57:20.629 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 12 0 2024-08-09T14:57:20.629 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 9 0 2024-08-09T14:57:20.629 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: 2 0 2024-08-09T14:57:20.629 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-08-09T14:57:20.629 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (2411 ms) 2024-08-09T14:57:20.629 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-08-09T14:57:20.629 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: x cursor=MIN 2024-08-09T14:57:20.629 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=1 cursor=11:02547ec2:::1:head 2024-08-09T14:57:20.629 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=10 cursor=11:52ea6a34:::10:head 2024-08-09T14:57:20.629 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=13 cursor=11:566253c9:::13:head 2024-08-09T14:57:20.630 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=7 cursor=11:5c6b0b28:::7:head 2024-08-09T14:57:20.630 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=14 cursor=11:62a1935d:::14:head 2024-08-09T14:57:20.630 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=0 cursor=11:6cac518f:::0:head 2024-08-09T14:57:20.630 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=15 cursor=11:863748b0:::15:head 2024-08-09T14:57:20.630 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=11 cursor=11:89d3ae78:::11:head 2024-08-09T14:57:20.630 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=5 cursor=11:b29083e3:::5:head 2024-08-09T14:57:20.630 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=8 cursor=11:bd63b0f1:::8:head 2024-08-09T14:57:20.630 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=6 cursor=11:c4fdafeb:::6:head 2024-08-09T14:57:20.630 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=3 cursor=11:cfc208b3:::3:head 2024-08-09T14:57:20.630 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=4 cursor=11:d83876eb:::4:head 2024-08-09T14:57:20.630 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=12 cursor=11:de5d7c5f:::12:head 2024-08-09T14:57:20.630 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=9 cursor=11:e960b815:::9:head 2024-08-09T14:57:20.631 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > oid=2 cursor=11:f905c69b:::2:head 2024-08-09T14:57:20.631 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: FIRST> seek to MIN oid=1 2024-08-09T14:57:20.631 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=1 cursor=11:02547ec2:::1:head 2024-08-09T14:57:20.631 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:02547ec2:::1:head 2024-08-09T14:57:20.631 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:02547ec2:::1:head -> 1 2024-08-09T14:57:20.631 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=10 cursor=11:52ea6a34:::10:head 2024-08-09T14:57:20.631 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:52ea6a34:::10:head 2024-08-09T14:57:20.631 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:52ea6a34:::10:head -> 10 2024-08-09T14:57:20.631 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=13 cursor=11:566253c9:::13:head 2024-08-09T14:57:20.631 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:566253c9:::13:head 2024-08-09T14:57:20.631 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:566253c9:::13:head -> 13 2024-08-09T14:57:20.631 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=7 cursor=11:5c6b0b28:::7:head 2024-08-09T14:57:20.631 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:5c6b0b28:::7:head 2024-08-09T14:57:20.631 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:5c6b0b28:::7:head -> 7 2024-08-09T14:57:20.683 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: cmd: Running main() from gmock_main.cc 2024-08-09T14:57:20.683 INFO:tasks.workunit.client.0.smithi155.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-08-09T14:57:20.683 INFO:tasks.workunit.client.0.smithi155.stdout: cmd: [----------] Global test environment set-up. 2024-08-09T14:57:20.683 INFO:tasks.workunit.client.0.smithi155.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-08-09T14:57:20.683 INFO:tasks.workunit.client.0.smithi155.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-08-09T14:57:20.683 INFO:tasks.workunit.client.0.smithi155.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (5576 ms) 2024-08-09T14:57:20.683 INFO:tasks.workunit.client.0.smithi155.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-08-09T14:57:20.683 INFO:tasks.workunit.client.0.smithi155.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (4319 ms) 2024-08-09T14:57:20.683 INFO:tasks.workunit.client.0.smithi155.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-08-09T14:57:20.683 INFO:tasks.workunit.client.0.smithi155.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (323241 ms) 2024-08-09T14:57:20.683 INFO:tasks.workunit.client.0.smithi155.stdout: cmd: [----------] 3 tests from NeoRadosCmd (333136 ms total) 2024-08-09T14:57:20.683 INFO:tasks.workunit.client.0.smithi155.stdout: cmd: 2024-08-09T14:57:20.683 INFO:tasks.workunit.client.0.smithi155.stdout: cmd: [----------] Global test environment tear-down 2024-08-09T14:57:20.684 INFO:tasks.workunit.client.0.smithi155.stdout: cmd: [==========] 3 tests from 1 test suite ran. (333136 ms total) 2024-08-09T14:57:20.684 INFO:tasks.workunit.client.0.smithi155.stdout: cmd: [ PASSED ] 3 tests. 2024-08-09T14:57:21.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-08-09T14:57:21.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-08-09T14:57:21.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-08-09T14:57:21.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-08-09T14:57:21.028 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-08-09T14:57:21.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-08-09T14:57:21.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-08-09T14:57:21.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (72846 ms) 2024-08-09T14:57:21.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-08-09T14:57:21.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (37535 ms) 2024-08-09T14:57:21.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-08-09T14:57:21.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (6333 ms) 2024-08-09T14:57:21.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-08-09T14:57:21.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (7682 ms) 2024-08-09T14:57:21.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-08-09T14:57:21.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (2025 ms) 2024-08-09T14:57:21.029 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-08-09T14:57:21.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (38665 ms) 2024-08-09T14:57:21.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-08-09T14:57:21.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (4319 ms) 2024-08-09T14:57:21.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-08-09T14:57:21.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (3976 ms) 2024-08-09T14:57:21.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-08-09T14:57:21.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (5990 ms) 2024-08-09T14:57:21.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-08-09T14:57:21.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (425 ms) 2024-08-09T14:57:21.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-08-09T14:57:21.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (12470 ms) 2024-08-09T14:57:21.030 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-08-09T14:57:21.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (3887 ms) 2024-08-09T14:57:21.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-08-09T14:57:21.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (11645 ms) 2024-08-09T14:57:21.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-08-09T14:57:21.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (106956 ms) 2024-08-09T14:57:21.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-08-09T14:57:21.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (759 ms) 2024-08-09T14:57:21.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-08-09T14:57:21.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (133 ms) 2024-08-09T14:57:21.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (315646 ms total) 2024-08-09T14:57:21.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: 2024-08-09T14:57:21.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-08-09T14:57:21.031 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (333555 ms total) 2024-08-09T14:57:21.032 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-08-09T14:57:25.529 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~75s 2024-08-09T14:57:28.743 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: Running main() from gmock_main.cc 2024-08-09T14:57:28.743 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-08-09T14:57:28.743 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [----------] Global test environment set-up. 2024-08-09T14:57:28.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-08-09T14:57:28.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-08-09T14:57:28.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (907 ms) 2024-08-09T14:57:28.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-08-09T14:57:28.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (328 ms) 2024-08-09T14:57:28.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-08-09T14:57:28.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (332366 ms) 2024-08-09T14:57:28.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-08-09T14:57:28.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:07.185920+0000 mon.b [INF] from='client.? 172.21.15.155:0/3374473416' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi155-41945-4"}]: dispatch 2024-08-09T14:57:28.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:07.759979+0000 mon.a [INF] Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-08-09T14:57:28.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:08.036014+0000 mon.c [INF] from='client.? 172.21.15.155:0/1932393983' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi155-41499-16"}]: dispatch 2024-08-09T14:57:28.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:08.407623+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi155-41945-4"}]: dispatch 2024-08-09T14:57:28.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:10.233409+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi155-41499-16"}]: dispatch 2024-08-09T14:57:28.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:10.258358+0000 mon.a [INF] from='client.? 172.21.15.155:0/3730101140' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi155-41701-6"}]': finished 2024-08-09T14:57:28.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:10.731212+0000 mon.a [WRN] Health check update: Degraded data redundancy: 34/400 objects degraded (8.500%), 26 pgs degraded (PG_DEGRADED) 2024-08-09T14:57:28.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:10.930300+0000 mon.a [INF] from='client.? 172.21.15.155:0/3730101140' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi155-41701-6", "tierpool": "test-rados-api-smithi155-41701-7"}]: dispatch 2024-08-09T14:57:28.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:11.140673+0000 mon.a [INF] from='client.4837 172.21.15.155:0/2877733599' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi155-42537-2"}]: dispatch 2024-08-09T14:57:28.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:11.860479+0000 mon.b [INF] from='client.? 172.21.15.155:0/3497978636' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi155-41617-10"}]: dispatch 2024-08-09T14:57:28.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:18.587796+0000 mon.a [WRN] Health check update: Degraded data redundancy: 46/513 objects degraded (8.967%), 34 pgs degraded, 111 pgs undersized (PG_DEGRADED) 2024-08-09T14:57:28.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:19.037450+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi155-41617-10"}]: dispatch 2024-08-09T14:57:28.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:19.521181+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi155-41945-4"}]': finished 2024-08-09T14:57:28.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:19.526522+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi155-41499-16"}]': finished 2024-08-09T14:57:29.792 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:05.035 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:57:29.952 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:05.192 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:57:30.091 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:05.334 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:57:30.565 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:20.772929+0000 mon.a [WRN] Health check update: 0 : oid=14 cursor=11:62a1935d:::14:head 2024-08-09T14:57:30.565 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:62a1935d:::14:head 2024-08-09T14:57:30.565 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:62a1935d:::14:head -> 14 2024-08-09T14:57:30.565 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=0 cursor=11:6cac518f:::0:head 2024-08-09T14:57:30.565 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:6cac518f:::0:head 2024-08-09T14:57:30.565 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:6cac518f:::0:head -> 0 2024-08-09T14:57:30.565 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=15 cursor=11:863748b0:::15:head 2024-08-09T14:57:30.565 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:863748b0:::15:head 2024-08-09T14:57:30.566 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:863748b0:::15:head -> 15 2024-08-09T14:57:30.566 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=11 cursor=11:89d3ae78:::11:head 2024-08-09T14:57:30.566 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:89d3ae78:::11:head 2024-08-09T14:57:30.566 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:89d3ae78:::11:head -> 11 2024-08-09T14:57:30.566 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=5 cursor=11:b29083e3:::5:head 2024-08-09T14:57:30.566 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:b29083e3:::5:head 2024-08-09T14:57:30.566 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:b29083e3:::5:head -> 5 2024-08-09T14:57:30.566 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=8 cursor=11:bd63b0f1:::8:head 2024-08-09T14:57:30.566 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:bd63b0f1:::8:head 2024-08-09T14:57:30.566 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:bd63b0f1:::8:head -> 8 2024-08-09T14:57:30.566 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=6 cursor=11:c4fdafeb:::6:head 2024-08-09T14:57:30.567 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:c4fdafeb:::6:head 2024-08-09T14:57:30.567 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:c4fdafeb:::6:head -> 6 2024-08-09T14:57:30.567 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=3 cursor=11:cfc208b3:::3:head 2024-08-09T14:57:30.567 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:cfc208b3:::3:head 2024-08-09T14:57:30.567 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:cfc208b3:::3:head -> 3 2024-08-09T14:57:30.567 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=4 cursor=11:d83876eb:::4:head 2024-08-09T14:57:30.567 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:d83876eb:::4:head 2024-08-09T14:57:30.567 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:d83876eb:::4:head -> 4 2024-08-09T14:57:30.567 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=12 cursor=11:de5d7c5f:::12:head 2024-08-09T14:57:30.567 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:de5d7c5f:::12:head 2024-08-09T14:57:30.567 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:de5d7c5f:::12:head -> 12 2024-08-09T14:57:30.567 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=9 cursor=11:e960b815:::9:head 2024-08-09T14:57:30.567 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:e960b815:::9:head 2024-08-09T14:57:30.568 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:e960b815:::9:head -> 9 2024-08-09T14:57:30.568 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : oid=2 cursor=11:f905c69b:::2:head 2024-08-09T14:57:30.568 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:f905c69b:::2:head 2024-08-09T14:57:30.568 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:f905c69b:::2:head -> 2 2024-08-09T14:57:30.568 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:02547ec2:::1:head 2024-08-09T14:57:30.568 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : cursor()=11:02547ec2:::1:head expected=11:02547ec2:::1:head 2024-08-09T14:57:30.568 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:02547ec2:::1:head -> 1 2024-08-09T14:57:30.568 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : entry=1 expected=1 2024-08-09T14:57:30.568 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:566253c9:::13:head 2024-08-09T14:57:30.568 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : cursor()=11:566253c9:::13:head expected=11:566253c9:::13:head 2024-08-09T14:57:30.568 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:566253c9:::13:head -> 13 2024-08-09T14:57:30.569 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : entry=13 expected=13 2024-08-09T14:57:30.569 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:52ea6a34:::10:head 2024-08-09T14:57:30.569 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : cursor()=11:52ea6a34:::10:head expected=11:52ea6a34:::10:head 2024-08-09T14:57:30.569 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:52ea6a34:::10:head -> 10 2024-08-09T14:57:30.569 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : entry=10 expected=10 2024-08-09T14:57:30.569 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:6cac518f:::0:head 2024-08-09T14:57:30.569 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : cursor()=11:6cac518f:::0:head expected=11:6cac518f:::0:head 2024-08-09T14:57:30.569 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:6cac518f:::0:head -> 0 2024-08-09T14:57:30.569 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : entry=0 expected=0 2024-08-09T14:57:30.569 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:62a1935d:::14:head 2024-08-09T14:57:30.569 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : cursor()=11:62a1935d:::14:head expected=11:62a1935d:::14:head 2024-08-09T14:57:30.569 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:62a1935d:::14:head -> 14 2024-08-09T14:57:30.569 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : entry=14 expected=14 2024-08-09T14:57:30.570 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:5c6b0b28:::7:head 2024-08-09T14:57:30.570 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : cursor()=11:5c6b0b28:::7:head expected=11:5c6b0b28:::7:head 2024-08-09T14:57:30.570 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:5c6b0b28:::7:head -> 7 2024-08-09T14:57:30.570 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : entry=7 expected=7 2024-08-09T14:57:30.570 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:f905c69b:::2:head 2024-08-09T14:57:30.570 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : cursor()=11:f905c69b:::2:head expected=11:f905c69b:::2:head 2024-08-09T14:57:30.570 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:f905c69b:::2:head -> 2 2024-08-09T14:57:30.570 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : entry=2 expected=2 2024-08-09T14:57:30.570 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : seek to 11:e960b815:::9:head 2024-08-09T14:57:30.570 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: : cursor()=11:e960b815:::9:head expected=11:e960b815:::9:head 2024-08-09T14:57:30.571 INFO:tasks.workunit.client.0.smithi155.stdout: api_list: > 11:e960b815:::9:head -> 9 2024-08-09T14:57:30.766 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:06.010 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:57:30.853 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:06.096 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:57:30.856 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:06.102 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:57:30.935 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:06.182 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:57:30.975 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:06.219 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:57:31.034 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:06.281 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:57:31.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~81s 2024-08-09T14:57:32.341 INFO:tasks.workunit.client.0.smithi155.stdout: a api_io: Running main() from gmock_main.cc 2024-08-09T14:57:32.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-08-09T14:57:32.341 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [----------] Global test environment set-up. 2024-08-09T14:57:32.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-08-09T14:57:32.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-08-09T14:57:32.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (67004 ms) 2024-08-09T14:57:32.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-08-09T14:57:32.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-08-09T14:57:32.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-08-09T14:57:32.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (10571 ms) 2024-08-09T14:57:32.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-08-09T14:57:32.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (27460 ms) 2024-08-09T14:57:32.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-08-09T14:57:32.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIo.Checksum (1611 ms) 2024-08-09T14:57:32.342 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-08-09T14:57:32.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (6344 ms) 2024-08-09T14:57:32.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-08-09T14:57:32.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (5006 ms) 2024-08-09T14:57:32.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-08-09T14:57:32.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (42791 ms) 2024-08-09T14:57:32.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-08-09T14:57:32.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (7116 ms) 2024-08-09T14:57:32.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-08-09T14:57:32.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4436 ms) 2024-08-09T14:57:32.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-08-09T14:57:32.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (5626 ms) 2024-08-09T14:57:32.343 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-08-09T14:57:32.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (11174 ms) 2024-08-09T14:57:32.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-08-09T14:57:32.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIo.RmXattr (23385 ms) 2024-08-09T14:57:32.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-08-09T14:57:32.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIo.XattrIter (13620 ms) 2024-08-09T14:57:32.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [----------] 14 tests from LibRadosIo (226145 ms total) 2024-08-09T14:57:32.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: 2024-08-09T14:57:32.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-08-09T14:57:32.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-08-09T14:57:32.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (19339 ms) 2024-08-09T14:57:32.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-08-09T14:57:32.344 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (4115 ms) 2024-08-09T14:57:32.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-08-09T14:57:32.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (7490 ms) 2024-08-09T14:57:32.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-08-09T14:57:32.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (8249 ms) 2024-08-09T14:57:32.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-08-09T14:57:32.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (11175 ms) 2024-08-09T14:57:32.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-08-09T14:57:32.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (25772 ms) 2024-08-09T14:57:32.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-08-09T14:57:32.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (226 ms) 2024-08-09T14:57:32.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-08-09T14:57:32.345 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (146 ms) 2024-08-09T14:57:32.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-08-09T14:57:32.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (1858 ms) 2024-08-09T14:57:32.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-08-09T14:57:32.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (754 ms) 2024-08-09T14:57:32.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [----------] 10 tests from LibRadosIoEC (79124 ms total) 2024-08-09T14:57:32.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: 2024-08-09T14:57:32.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [----------] Global test environment tear-down 2024-08-09T14:57:32.346 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [==========] 24 tests from 2 test suites ran. (345149 ms total) 2024-08-09T14:57:32.347 INFO:tasks.workunit.client.0.smithi155.stdout: api_io: [ PASSED ] 24 tests. 2024-08-09T14:57:32.726 INFO:tasks.workunit.client.0.smithi155.stdout:slow ops, oldest one blocked for 84 sec, osd.3 has slow ops (SLOW_OPS) 2024-08-09T14:57:32.726 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:20.801982+0000 mon.c [INF] from='client.? 172.21.15.155:0/4098620087' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi155-41823-7"}]: dispatch 2024-08-09T14:57:32.726 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:20.813462+0000 mon.c [INF] from='client.? 172.21.15.155:0/1932393983' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi155-41499-16"}]: dispatch 2024-08-09T14:57:32.727 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:21.052507+0000 mon.b [INF] from='client.? 172.21.15.155:0/3374473416' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi155-41945-4", "tierpool": "test-rados-api-smithi155-41945-6"}]: dispatch 2024-08-09T14:57:32.727 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:22.346756+0000 mon.a [WRN] Health check update: 13 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-09T14:57:32.727 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:23.237384+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi155-41945-4", "tierpool": "test-rados-api-smithi155-41945-6"}]: dispatch 2024-08-09T14:57:32.727 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:23.638109+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi155-41823-7"}]: dispatch 2024-08-09T14:57:32.727 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:23.641772+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi155-41499-16"}]: dispatch 2024-08-09T14:57:32.727 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:23.686409+0000 mon.a [INF] from='client.? 172.21.15.155:0/3730101140' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi155-41701-6", "tierpool": "test-rados-api-smithi155-41701-7"}]': finished 2024-08-09T14:57:32.727 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:23.688236+0000 mon.a [INF] from='client.4837 172.21.15.155:0/2877733599' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi155-42537-2"}]': finished 2024-08-09T14:57:32.727 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:23.689940+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi155-41617-10"}]': finished 2024-08-09T14:57:32.727 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:24.326734+0000 mon.a [INF] from='client.4859 172.21.15.155:0/989749044' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi155-41675-10"}]: dispatch 2024-08-09T14:57:32.727 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:24.484135+0000 mon.a [INF] from='client.4837 172.21.15.155:0/2877733599' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi155-42537-2"}]: dispatch 2024-08-09T14:57:32.727 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:23.679045+0000 mon.b [INF] from='client.? 172.21.15.155:0/3874951363' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-08-09T14:57:32.727 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:24.738330+0000 mon.a [INF] Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-08-09T14:57:32.728 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:25.209285+0000 client.admin [INF] onexx 2024-08-09T14:57:32.728 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:25.252611+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-08-09T14:57:32.728 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:25.485748+0000 mon.a [INF] from='client.4999 172.21.15.155:0/1489524443' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi155-41515-22"}]: dispatch 2024-08-09T14:57:34.020 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:25.540536+0000 mon.b [INF] from='client.? 172.21.15.155:0/3497978636' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLoc api_lock: Running main() from gmock_main.cc 2024-08-09T14:57:34.020 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-08-09T14:57:34.021 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [----------] Global test environment set-up. 2024-08-09T14:57:34.021 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-08-09T14:57:34.021 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-08-09T14:57:34.021 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (100787 ms) 2024-08-09T14:57:34.021 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-08-09T14:57:34.021 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLock.LockShared (3735 ms) 2024-08-09T14:57:34.021 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-08-09T14:57:34.021 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (17899 ms) 2024-08-09T14:57:34.021 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-08-09T14:57:34.021 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (4453 ms) 2024-08-09T14:57:34.022 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-08-09T14:57:34.022 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (5764 ms) 2024-08-09T14:57:34.022 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-08-09T14:57:34.022 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLock.Unlock (42692 ms) 2024-08-09T14:57:34.022 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-08-09T14:57:34.022 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (51208 ms) 2024-08-09T14:57:34.022 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-08-09T14:57:34.022 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (40135 ms) 2024-08-09T14:57:34.022 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [----------] 8 tests from LibRadosLock (266673 ms total) 2024-08-09T14:57:34.022 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: 2024-08-09T14:57:34.023 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-08-09T14:57:34.023 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-08-09T14:57:34.023 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (23152 ms) 2024-08-09T14:57:34.023 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-08-09T14:57:34.023 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (6808 ms) 2024-08-09T14:57:34.023 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-08-09T14:57:34.023 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (4320 ms) 2024-08-09T14:57:34.023 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-08-09T14:57:34.023 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (3934 ms) 2024-08-09T14:57:34.023 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-08-09T14:57:34.023 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (290 ms) 2024-08-09T14:57:34.023 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-08-09T14:57:34.023 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (4143 ms) 2024-08-09T14:57:34.024 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-08-09T14:57:34.024 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (270 ms) 2024-08-09T14:57:34.024 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-08-09T14:57:34.024 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (3014 ms) 2024-08-09T14:57:34.024 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (45931 ms total) 2024-08-09T14:57:34.024 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: 2024-08-09T14:57:34.024 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [----------] Global test environment tear-down 2024-08-09T14:57:34.024 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (346760 ms total) 2024-08-09T14:57:34.024 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock: [ PASSED ] 16 tests. 2024-08-09T14:57:37.948 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~87s 2024-08-09T14:57:38.263 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-08-09T14:57:38.244+0000 ffc8640 -1 mon.c@2(peon).paxos(paxos updating c 252..772) lease_expire from mon.0 v2:172.21.15.155:3300/0 is 1.005832791s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2024-08-09T14:57:41.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-08-09T14:57:41.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-08-09T14:57:41.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-08-09T14:57:41.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-08-09T14:57:41.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: seed 41617 2024-08-09T14:57:41.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-08-09T14:57:41.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (15183 ms) 2024-08-09T14:57:41.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-08-09T14:57:41.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (58381 ms) 2024-08-09T14:57:41.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-08-09T14:57:41.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (49755 ms) 2024-08-09T14:57:41.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-08-09T14:57:41.396 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (2916 ms) 2024-08-09T14:57:41.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-08-09T14:57:41.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (4610 ms) 2024-08-09T14:57:41.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-08-09T14:57:41.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (8368 ms) 2024-08-09T14:57:41.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-08-09T14:57:41.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (45213 ms) 2024-08-09T14:57:41.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-08-09T14:57:41.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (15113 ms) 2024-08-09T14:57:41.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (199539 ms total) 2024-08-09T14:57:41.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: 2024-08-09T14:57:41.397 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-08-09T14:57:41.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-08-09T14:57:41.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (9755 ms) 2024-08-09T14:57:41.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-08-09T14:57:41.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (14671 ms) 2024-08-09T14:57:41.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-08-09T14:57:41.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (34148 ms) 2024-08-09T14:57:41.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-08-09T14:57:41.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (47648 ms) 2024-08-09T14:57:41.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-08-09T14:57:41.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (606 ms) 2024-08-09T14:57:41.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-08-09T14:57:41.398 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (334 ms) 2024-08-09T14:57:41.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-08-09T14:57:41.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (204 ms) 2024-08-09T14:57:41.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-08-09T14:57:41.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (124 ms) 2024-08-09T14:57:41.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (107490 ms total) 2024-08-09T14:57:41.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: 2024-08-09T14:57:41.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-08-09T14:57:41.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (354150 ms total) 2024-08-09T14:57:41.399 INFO:tasks.workunit.client.0.smithi155.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-08-09T14:57:44.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~93s 2024-08-09T14:57:45.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-08-09T14:57:45.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-08-09T14:57:45.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-08-09T14:57:45.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-08-09T14:57:45.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: seed 41823 2024-08-09T14:57:45.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-08-09T14:57:45.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (37693 ms) 2024-08-09T14:57:45.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-08-09T14:57:45.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (11881 ms) 2024-08-09T14:57:45.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-08-09T14:57:45.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (111 ms) 2024-08-09T14:57:45.155 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-08-09T14:57:45.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (2937 ms) 2024-08-09T14:57:45.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-08-09T14:57:45.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (91408 ms) 2024-08-09T14:57:45.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (144030 ms total) 2024-08-09T14:57:45.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: 2024-08-09T14:57:45.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-08-09T14:57:45.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-08-09T14:57:45.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (99093 ms) 2024-08-09T14:57:45.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-08-09T14:57:45.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (175 ms) 2024-08-09T14:57:45.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-08-09T14:57:45.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (56625 ms) 2024-08-09T14:57:45.156 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-08-09T14:57:45.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (9385 ms) 2024-08-09T14:57:45.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (165278 ms total) 2024-08-09T14:57:45.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: 2024-08-09T14:57:45.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-08-09T14:57:45.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (357820 ms total) 2024-08-09T14:57:45.157 INFO:tasks.workunit.client.0.smithi155.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-08-09T14:57:46.477 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_1_[42340]: starting. 2024-08-09T14:57:46.477 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_1_[42340]: creating pool ceph_test_rados_list_parallel.smithi155-42268 2024-08-09T14:57:46.478 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_1_[42340]: created object 0... 2024-08-09T14:57:46.478 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_1_[42340]: created object 25... 2024-08-09T14:57:46.478 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_1_[42340]: created object 49... 2024-08-09T14:57:46.478 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_1_[42340]: finishing. 2024-08-09T14:57:46.478 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_1_[42340]: shutting down. 2024-08-09T14:57:46.478 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_2_[42341]: starting. 2024-08-09T14:57:46.478 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_2_[42341]: listing objects. 2024-08-09T14:57:46.478 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_2_[42341]: listed object 0... 2024-08-09T14:57:46.478 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_2_[42341]: listed object 25... 2024-08-09T14:57:46.478 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_2_[42341]: saw 50 objects 2024-08-09T14:57:46.478 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_2_[42341]: shutting down. 2024-08-09T14:57:46.478 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:57:46.479 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_3_[67164]: starting. 2024-08-09T14:57:46.479 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_3_[67164]: creating pool ceph_test_rados_list_parallel.smithi155-42268 2024-08-09T14:57:46.479 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_3_[67164]: created object 0... 2024-08-09T14:57:46.479 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_3_[67164]: created object 25... 2024-08-09T14:57:46.479 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_3_[67164]: created object 49... 2024-08-09T14:57:46.479 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_3_[67164]: finishing. 2024-08-09T14:57:46.479 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_3_[67164]: shutting down. 2024-08-09T14:57:46.479 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:57:46.479 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_4_[67165]: starting. 2024-08-09T14:57:46.479 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_4_[67165]: listing objects. 2024-08-09T14:57:46.479 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_4_[67165]: listed object 0... 2024-08-09T14:57:46.479 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_4_[67165]: listed object 25... 2024-08-09T14:57:46.479 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_4_[67165]: saw 45 objects 2024-08-09T14:57:46.480 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_4_[67165]: shutting down. 2024-08-09T14:57:46.480 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:57:46.480 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_5_[67166]: starting. 2024-08-09T14:57:46.480 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_5_[67166]: removed 25 objects... 2024-08-09T14:57:46.480 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_5_[67166]: removed half of the objects 2024-08-09T14:57:46.480 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_5_[67166]: removed 50 objects... 2024-08-09T14:57:46.480 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_5_[67166]: removed 50 objects 2024-08-09T14:57:46.480 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_5_[67166]: shutting down. 2024-08-09T14:57:46.480 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:57:46.480 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:57:46.480 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_6_[79057]: starting. 2024-08-09T14:57:46.480 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_6_[79057]: creating pool ceph_test_rados_list_parallel.smithi155-42268 2024-08-09T14:57:46.481 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_6_[79057]: created object 0... 2024-08-09T14:57:46.481 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_6_[79057]: created object 25... 2024-08-09T14:57:46.481 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_6_[79057]: created object 49... 2024-08-09T14:57:46.481 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_6_[79057]: finishing. 2024-08-09T14:57:46.481 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_6_[79057]: shutting down. 2024-08-09T14:57:46.481 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:57:46.481 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:57:46.481 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_7_[79058]: starting. 2024-08-09T14:57:46.481 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_7_[79058]: listing objects. 2024-08-09T14:57:46.481 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_7_[79058]: listed object 0... 2024-08-09T14:57:46.481 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_7_[79058]: listed object 25... 2024-08-09T14:57:46.481 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_7_[79058]: listed object 50... 2024-08-09T14:57:46.481 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_7_[79058]: saw 56 objects 2024-08-09T14:57:46.482 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_7_[79058]: shutting down. 2024-08-09T14:57:46.482 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:57:46.482 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:57:46.482 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_8_[79059]: starting. 2024-08-09T14:57:46.482 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_8_[79059]: added 25 objects... 2024-08-09T14:57:46.482 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_8_[79059]: added half of the objects 2024-08-09T14:57:46.482 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_8_[79059]: added 50 objects... 2024-08-09T14:57:46.482 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_8_[79059]: added 50 objects 2024-08-09T14:57:46.482 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_8_[79059]: shutting down. 2024-08-09T14:57:46.482 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:57:46.482 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:57:48.601 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: **********************adosIoECPP.RoundTripPP2 (202 ms) 2024-08-09T14:57:48.601 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-08-09T14:57:48.601 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (1552 ms) 2024-08-09T14:57:48.601 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-08-09T14:57:48.601 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (273 ms) 2024-08-09T14:57:48.601 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-08-09T14:57:48.601 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (330 ms) 2024-08-09T14:57:48.601 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-08-09T14:57:48.602 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (355 ms) 2024-08-09T14:57:48.602 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-08-09T14:57:48.602 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (168 ms) 2024-08-09T14:57:48.602 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-08-09T14:57:48.602 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (152 ms) 2024-08-09T14:57:48.602 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-08-09T14:57:48.602 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (2526 ms) 2024-08-09T14:57:48.602 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-08-09T14:57:48.602 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (2011 ms) 2024-08-09T14:57:48.602 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-08-09T14:57:48.602 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (221 ms) 2024-08-09T14:57:48.602 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-08-09T14:57:48.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (282 ms) 2024-08-09T14:57:48.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-08-09T14:57:48.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (203 ms) 2024-08-09T14:57:48.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-08-09T14:57:48.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (193 ms) 2024-08-09T14:57:48.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (54407 ms total) 2024-08-09T14:57:48.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: 2024-08-09T14:57:48.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [----------] Global test environment tear-down 2024-08-09T14:57:48.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (361404 ms total) 2024-08-09T14:57:48.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-08-09T14:57:49.452 INFO:tasks.workunit.client.0.smithi155.stdout:kECPP_smithi155-41617-10"}]: dispatch 2024-08-09T14:57:49.452 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:25.837220+0000 mon.c [INF] from='client.? 172.21.15.155:0/3202520998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi155-41460-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-09T14:57:49.453 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:26.100133+0000 mon.a [WRN] Health check update: Degraded data redundancy: 45/520 objects degraded (8.654%), 28 pgs degraded, 146 pgs undersized (PG_DEGRADED) 2024-08-09T14:57:49.453 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:27.174642+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi155-41617-10"}]: dispatch 2024-08-09T14:57:49.453 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:27.630191+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi155-41460-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-09T14:57:49.453 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:27.661331+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi155-41945-4", "tierpool": "test-rados-api-smithi155-41945-6"}]': finished 2024-08-09T14:57:49.453 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:27.663921+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi155-41823-7"}]': finished 2024-08-09T14:57:49.453 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:27.665983+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi155-41499-16"}]': finished 2024-08-09T14:57:49.453 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:27.668197+0000 mon.a [INF] from='client.4859 172.21.15.155:0/989749044' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi155-41675-10"}]': finished 2024-08-09T14:57:49.453 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:27.669607+0000 mon.a [INF] from='client.4837 172.21.15.155:0/2877733599' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi155-42537-2"}]': finished 2024-08-09T14:57:49.453 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:28.321608+0000 mon.a [INF] from='client.4859 172.21.15.155:0/989749044' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi155-41675-10"}]: dispatch 2024-08-09T14:57:49.453 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:31.373269+0000 mon.c [INF] from='client.? 172.21.15.155:0/1864292104' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi155-42537-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-08-09T14:57:49.453 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:33.376659+0000 mon.b [INF] from='client.? 172.21.15.155:0/3874951363' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-08-09T14:57:49.453 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:33.570828+0000 mon.a [INF] from='client.4999 172.21.15.155:0/1489524443' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi155-41515-22"}]: dispatch 2024-08-09T14:57:49.454 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:33.579576+0000 mon.b [INF] from='client.? 172.21.15.155:0/1998075150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi155-41975-12"}]: dispatch 2024-08-09T14:57:49.454 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:36.155992+0000 mon.a [WRN] Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-09T14:57:49.454 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:40.344870+0000 mon.a [WRN] Health check update: Degraded data redundancy: 30/250 objects degraded (12.000%), 21 pgs degraded, 122 pgs undersized (PG_DEGRADED) 2024-08-09T14:57:49.454 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:40.967437+0000 client.admin [INF] twoxx 2024-08-09T14:57:49.454 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:40.985937+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-08-09T14:57:50.367 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~99s 2024-08-09T14:57:51.584 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-eout/0 2024-08-09T14:57:51.584 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: trying... 2024-08-09T14:57:51.584 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: handle_notify cookie 94471279659024 notify_id 884763262987 notifier_gid 4804 2024-08-09T14:57:51.584 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: timed out 2024-08-09T14:57:51.584 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: flushing 2024-08-09T14:57:51.584 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: flushed 2024-08-09T14:57:51.585 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (9257 ms) 2024-08-09T14:57:51.585 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-08-09T14:57:51.585 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: trying... 2024-08-09T14:57:51.585 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: handle_notify cookie 94471279659024 notify_id 919123001356 notifier_gid 4804 2024-08-09T14:57:51.585 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: timed out 2024-08-09T14:57:51.585 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: flushing 2024-08-09T14:57:51.585 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: flushed 2024-08-09T14:57:51.585 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (8538 ms) 2024-08-09T14:57:51.585 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-08-09T14:57:51.585 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: List watches 2024-08-09T14:57:51.585 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: notify2 2024-08-09T14:57:51.585 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: handle_notify cookie 94471280519584 notify_id 1030792151040 notifier_gid 4804 2024-08-09T14:57:51.586 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: handle_notify cookie 94471280519584 notify_id 1099511627778 notifier_gid 4804 2024-08-09T14:57:51.586 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: notify2 done 2024-08-09T14:57:51.586 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: watch_check 2024-08-09T14:57:51.586 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: unwatch2 2024-08-09T14:57:51.586 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: flushing 2024-08-09T14:57:51.586 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: done 2024-08-09T14:57:51.586 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (57866 ms) 2024-08-09T14:57:51.586 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-08-09T14:57:51.586 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: List watches 2024-08-09T14:57:51.586 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: notify2 2024-08-09T14:57:51.586 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: handle_notify cookie 94471280519584 notify_id 1232655613955 notifier_gid 4804 2024-08-09T14:57:51.586 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: notify2 done 2024-08-09T14:57:51.586 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: watch_check 2024-08-09T14:57:51.587 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: unwatch2 2024-08-09T14:57:51.587 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: flushing 2024-08-09T14:57:51.587 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: done 2024-08-09T14:57:51.587 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (31893 ms) 2024-08-09T14:57:51.587 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (239095 ms total) 2024-08-09T14:57:51.587 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: 2024-08-09T14:57:51.587 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-08-09T14:57:51.587 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (364194 ms total) 2024-08-09T14:57:51.587 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-08-09T14:57:56.498 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-08-09T14:57:56.489+0000 ffc8640 -1 mon.c@2(peon).paxos(paxos updating c 252..779) lease_expire from mon.0 v2:172.21.15.155:3300/0 is 0.021453490s seconds in the past; mons are probably laggy (or possibly clocks are too skewed) 2024-08-09T14:57:56.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~106s 2024-08-09T14:57:58.394 INFO:tasks.workunit.client.0.smithi155.stdout:08-09T14:57:41.005900+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi155-41975-12"}]: dispatch 2024-08-09T14:57:58.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:41.206616+0000 mon.a [WRN] Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-09T14:57:58.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:41.337975+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi155-42537-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-08-09T14:57:58.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:41.501665+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi155-41945-6", "pool2": "test-rados-api-smithi155-41945-6", "yes_i_really_really_mean_it": true}]': finished 2024-08-09T14:57:58.394 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:41.503676+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi155-41823-7"}]': finished 2024-08-09T14:57:58.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:43.042565+0000 mon.b [INF] from='client.? 172.21.15.155:0/1998075150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi155-41975-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-08-09T14:57:58.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:43.460849+0000 mon.b [INF] from='client.? 172.21.15.155:0/34185881' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi155-41478-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-09T14:57:58.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:45.730286+0000 mon.a [INF] from='client.4999 172.21.15.155:0/1489524443' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi155-41515-22"}]': finished 2024-08-09T14:57:58.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:45.745326+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi155-42537-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-09T14:57:58.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:45.788547+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-08-09T14:57:58.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:46.047468+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi155-41975-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-08-09T14:57:58.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:46.116949+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi155-41478-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-09T14:57:58.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:46.237051+0000 mon.a [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-09T14:57:58.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:46.632957+0000 mon.a [INF] from='client.? 172.21.15.155:0/3730101140' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi155-41701-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-08-09T14:57:58.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:47.605565+0000 mon.c [INF] from='client.? 172.21.15.155:0/1864292104' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi155-42537-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi155-42537-3"}]: dispatch 2024-08-09T14:57:58.395 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:48.417877+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on back (longest 20930.802ms) (OSD_SLOW_PING_TIME_BACK) 2024-08-09T14:58:02.788 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~112s 2024-08-09T14:58:04.448 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:48.418649+0000 mon.a [WRN] Health check update: Slow OSD heartbeats on front (longest 22189 delete_pools_parallel: process_1_[42346]: starting. 2024-08-09T14:58:04.448 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_1_[42346]: creating pool ceph_test_rados_delete_pools_parallel.smithi155-42334 2024-08-09T14:58:04.449 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_1_[42346]: created object 0... 2024-08-09T14:58:04.449 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_1_[42346]: created object 25... 2024-08-09T14:58:04.449 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_1_[42346]: created object 49... 2024-08-09T14:58:04.449 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_1_[42346]: finishing. 2024-08-09T14:58:04.449 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_1_[42346]: shutting down. 2024-08-09T14:58:04.449 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_2_[42348]: starting. 2024-08-09T14:58:04.449 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_2_[42348]: deleting pool ceph_test_rados_delete_pools_parallel.smithi155-42334 2024-08-09T14:58:04.449 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_2_[42348]: shutting down. 2024-08-09T14:58:04.449 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: ******************************* 2024-08-09T14:58:04.449 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_3_[76356]: starting. 2024-08-09T14:58:04.449 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_3_[76356]: creating pool ceph_test_rados_delete_pools_parallel.smithi155-42334 2024-08-09T14:58:04.449 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_3_[76356]: created object 0... 2024-08-09T14:58:04.449 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_3_[76356]: created object 25... 2024-08-09T14:58:04.450 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_3_[76356]: created object 49... 2024-08-09T14:58:04.450 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_3_[76356]: finishing. 2024-08-09T14:58:04.450 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_3_[76356]: shutting down. 2024-08-09T14:58:04.450 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: ******************************* 2024-08-09T14:58:04.450 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_5_[76361]: starting. 2024-08-09T14:58:04.450 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_5_[76361]: listing objects. 2024-08-09T14:58:04.450 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_5_[76361]: listed object 0... 2024-08-09T14:58:04.450 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_5_[76361]: listed object 25... 2024-08-09T14:58:04.450 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_5_[76361]: saw 50 objects 2024-08-09T14:58:04.450 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_5_[76361]: shutting down. 2024-08-09T14:58:04.450 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: ******************************* 2024-08-09T14:58:04.451 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_4_[76357]: starting. 2024-08-09T14:58:04.451 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_4_[76357]: deleting pool ceph_test_rados_delete_pools_parallel.smithi155-42334 2024-08-09T14:58:04.451 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: process_4_[76357]: shutting down. 2024-08-09T14:58:04.451 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: ******************************* 2024-08-09T14:58:04.451 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: ******************************* 2024-08-09T14:58:04.451 INFO:tasks.workunit.client.0.smithi155.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-08-09T14:58:04.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-08-09T14:58:04.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-08-09T14:58:04.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-08-09T14:58:04.472 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-08-09T14:58:04.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ OK ] LibRadosMisc.Exec (49151 ms) 2024-08-09T14:58:04.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-08-09T14:58:04.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (155908 ms) 2024-08-09T14:58:04.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-08-09T14:58:04.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (10116 ms) 2024-08-09T14:58:04.473 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-08-09T14:58:04.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ OK ] LibRadosMisc.Applications (6727 ms) 2024-08-09T14:58:04.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-08-09T14:58:04.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-08-09T14:58:04.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-08-09T14:58:04.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-08-09T14:58:04.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-08-09T14:58:04.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (74285 ms) 2024-08-09T14:58:04.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [----------] 8 tests from LibRadosMisc (296187 ms total) 2024-08-09T14:58:04.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: 2024-08-09T14:58:04.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [----------] Global test environment tear-down 2024-08-09T14:58:04.474 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (377205 ms total) 2024-08-09T14:58:04.475 INFO:tasks.workunit.client.0.smithi155.stdout: api_misc: [ PASSED ] 12 tests. 2024-08-09T14:58:06.602 INFO:tasks.workunit.client.0.smithi155.stdout:.906ms) (OSD_SLOW_PING_TIME_FRONT) 2024-08-09T14:58:06.602 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:48.419328+0000 mon.a [WRN] Health check update: Degraded data redundancy: 32/312 objects degraded (10.256%), 18 pgs degraded, 109 pgs undersized (PG_DEGRADED) 2024-08-09T14:58:06.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:49.363473+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi155-41975-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-08-09T14:58:06.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:49.368467+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi155-41478-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-09T14:58:06.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:49.684614+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi155-42537-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi155-42537-3"}]: dispatch 2024-08-09T14:58:06.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:49.315831+0000 mon.b [INF] from='client.? 172.21.15.155:0/3874951363' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-08-09T14:58:06.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:50.898442+0000 mon.b [INF] from='client.? 172.21.15.155:0/1998075150' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi155-41975-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi155-41975-12"}]: dispatch 2024-08-09T14:58:06.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:51.105687+0000 client.admin [INF] threexx 2024-08-09T14:58:06.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:51.174233+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-08-09T14:58:06.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:51.811572+0000 mon.a [WRN] Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-09T14:58:06.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:52.783647+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi155-41975-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi155-41975-12"}]: dispatch 2024-08-09T14:58:06.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: got: 2024-08-09T14:57:57.229128+0000 mon.a [INF] from='client.? 172.21.15.155:0/3730101140' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi155-41701-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-09T14:58:06.603 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (45599 ms) 2024-08-09T14:58:06.604 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (379200 ms total) 2024-08-09T14:58:06.604 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: 2024-08-09T14:58:06.604 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [----------] Global test environment tear-down 2024-08-09T14:58:06.604 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (379200 ms total) 2024-08-09T14:58:06.604 INFO:tasks.workunit.client.0.smithi155.stdout: api_cmd: [ PASSED ] 4 tests. 2024-08-09T14:58:07.022 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:42.265 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:07.030 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:42.274 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:07.093 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:42.336 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:07.098 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:42.345 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:07.100 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:42.344 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:07.104 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:42.351 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:07.118 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:08:42.361 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:07.168 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:08:42.411 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:07.227 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:08:42.474 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:08.998 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~118s 2024-08-09T14:58:15.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~124s 2024-08-09T14:58:19.743 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-08-09T14:58:19.743 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-08-09T14:58:19.743 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-08-09T14:58:19.743 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-08-09T14:58:19.743 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-08-09T14:58:19.743 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify_test_cb 2024-08-09T14:58:19.743 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (40359 ms) 2024-08-09T14:58:19.743 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-08-09T14:58:19.743 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-08-09T14:58:19.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94507551719760 err -107 2024-08-09T14:58:19.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (45197 ms) 2024-08-09T14:58:19.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-08-09T14:58:19.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-08-09T14:58:19.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94507551719760 err -107 2024-08-09T14:58:19.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (66134 ms) 2024-08-09T14:58:19.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-08-09T14:58:19.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify2_test_cb from 4480 notify_id 768799145985 cookie 94507551731600 2024-08-09T14:58:19.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (51804 ms) 2024-08-09T14:58:19.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-08-09T14:58:19.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify2_test_cb from 4480 notify_id 884763262976 cookie 94507552408672 2024-08-09T14:58:19.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (17801 ms) 2024-08-09T14:58:19.744 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-08-09T14:58:19.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify2_test_cb from 4480 notify_id 932007903233 cookie 94507552408672 2024-08-09T14:58:19.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify2_test_cb from 4480 notify_id 936302870530 cookie 94507552408672 2024-08-09T14:58:19.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (12314 ms) 2024-08-09T14:58:19.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-08-09T14:58:19.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify2_test_cb from 4480 notify_id 1056561954817 cookie 94507552429888 2024-08-09T14:58:19.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify2_test_cb from 4480 notify_id 1056561954817 cookie 94507552444912 2024-08-09T14:58:19.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (46134 ms) 2024-08-09T14:58:19.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-08-09T14:58:19.745 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify2_test_cb from 4480 notify_id 1228360646657 cookie 94507552444912 2024-08-09T14:58:19.746 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify2_test_cb from 4480 notify_id 1232655613954 cookie 94507552444912 2024-08-09T14:58:19.746 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (28015 ms) 2024-08-09T14:58:19.746 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-08-09T14:58:19.746 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-08-09T14:58:19.746 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94507552444912 err -107 2024-08-09T14:58:19.746 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify2_test_cb from 4480 notify_id 1245540515854 cookie 94507552444912 2024-08-09T14:58:19.746 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (10784 ms) 2024-08-09T14:58:19.746 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-08-09T14:58:19.746 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-08-09T14:58:19.746 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94507552444912 err -107 2024-08-09T14:58:19.746 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (2149 ms) 2024-08-09T14:58:19.747 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (320692 ms total) 2024-08-09T14:58:19.747 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: 2024-08-09T14:58:19.747 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-08-09T14:58:19.747 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-08-09T14:58:19.747 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: watch_notify_test_cb 2024-08-09T14:58:19.747 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (13646 ms) 2024-08-09T14:58:19.747 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (13646 ms total) 2024-08-09T14:58:19.747 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: 2024-08-09T14:58:19.748 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-08-09T14:58:19.748 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (392346 ms total) 2024-08-09T14:58:19.748 INFO:tasks.workunit.client.0.smithi155.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-08-09T14:58:21.417 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~130s 2024-08-09T14:58:27.376 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:02.619 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:27.379 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:02.623 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:27.432 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:02.675 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:27.436 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:02.680 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:27.441 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:02.688 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:27.445 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:02.691 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:27.466 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:02.708 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:27.537 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:02.784 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:27.541 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:02.784 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:27.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~137s 2024-08-09T14:58:33.836 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~143s 2024-08-09T14:58:40.046 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~149s 2024-08-09T14:58:46.256 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~155s 2024-08-09T14:58:47.486 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:22.729 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:47.488 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:22.732 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:47.559 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:22.803 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:47.562 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:22.809 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:47.563 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:22.807 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:47.564 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:22.810 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:47.600 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:22.843 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:47.651 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:22.895 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:47.658 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:22.904 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:58:51.603 INFO:tasks.workunit.client.0.smithi155.stdout: io: Running main() from gmock_main.cc 2024-08-09T14:58:51.603 INFO:tasks.workunit.client.0.smithi155.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-08-09T14:58:51.603 INFO:tasks.workunit.client.0.smithi155.stdout: io: [----------] Global test environment set-up. 2024-08-09T14:58:51.603 INFO:tasks.workunit.client.0.smithi155.stdout: io: [----------] 14 tests from NeoRadosIo 2024-08-09T14:58:51.603 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-08-09T14:58:51.603 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ OK ] NeoRadosIo.Limits (41847 ms) 2024-08-09T14:58:51.603 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-08-09T14:58:51.603 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (65795 ms) 2024-08-09T14:58:51.603 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-08-09T14:58:51.603 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ OK ] NeoRadosIo.ReadOp (56718 ms) 2024-08-09T14:58:51.604 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-08-09T14:58:51.604 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ OK ] NeoRadosIo.SparseRead (10327 ms) 2024-08-09T14:58:51.604 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-08-09T14:58:51.604 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ OK ] NeoRadosIo.RoundTrip (10321 ms) 2024-08-09T14:58:51.604 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-08-09T14:58:51.604 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (19236 ms) 2024-08-09T14:58:51.604 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-08-09T14:58:51.604 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (10884 ms) 2024-08-09T14:58:51.604 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-08-09T14:58:51.604 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (19553 ms) 2024-08-09T14:58:51.604 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-08-09T14:58:51.604 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (56492 ms) 2024-08-09T14:58:51.604 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-08-09T14:58:51.604 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ OK ] NeoRadosIo.Trunc (32034 ms) 2024-08-09T14:58:51.605 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-08-09T14:58:51.605 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ OK ] NeoRadosIo.Remove (35689 ms) 2024-08-09T14:58:51.605 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-08-09T14:58:51.605 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (28899 ms) 2024-08-09T14:58:51.605 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-08-09T14:58:51.605 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ OK ] NeoRadosIo.RmXattr (19139 ms) 2024-08-09T14:58:51.605 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-08-09T14:58:51.605 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ OK ] NeoRadosIo.GetXattrs (17013 ms) 2024-08-09T14:58:51.606 INFO:tasks.workunit.client.0.smithi155.stdout: io: [----------] 14 tests from NeoRadosIo (423947 ms total) 2024-08-09T14:58:51.606 INFO:tasks.workunit.client.0.smithi155.stdout: io: 2024-08-09T14:58:51.606 INFO:tasks.workunit.client.0.smithi155.stdout: io: [----------] Global test environment tear-down 2024-08-09T14:58:51.606 INFO:tasks.workunit.client.0.smithi155.stdout: io: [==========] 14 tests from 1 test suite ran. (423947 ms total) 2024-08-09T14:58:51.606 INFO:tasks.workunit.client.0.smithi155.stdout: io: [ PASSED ] 14 tests. 2024-08-09T14:58:52.466 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~161s 2024-08-09T14:58:54.795 INFO:tasks.workunit.client.0.smithi155.stdout:********* 2024-08-09T14:58:54.796 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_9_[88991]: starting. 2024-08-09T14:58:54.796 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_9_[88991]: creating pool ceph_test_rados_list_parallel.smithi155-42268 2024-08-09T14:58:54.796 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_9_[88991]: created object 0... 2024-08-09T14:58:54.796 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_9_[88991]: created object 25... 2024-08-09T14:58:54.796 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_9_[88991]: created object 49... 2024-08-09T14:58:54.797 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_9_[88991]: finishing. 2024-08-09T14:58:54.797 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_9_[88991]: shutting down. 2024-08-09T14:58:54.797 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.797 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.797 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.797 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_10_[88992]: starting. 2024-08-09T14:58:54.798 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_10_[88992]: listing objects. 2024-08-09T14:58:54.798 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_10_[88992]: listed object 0... 2024-08-09T14:58:54.798 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_10_[88992]: listed object 25... 2024-08-09T14:58:54.798 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_10_[88992]: listed object 50... 2024-08-09T14:58:54.798 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_10_[88992]: listed object 75... 2024-08-09T14:58:54.798 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_10_[88992]: saw 95 objects 2024-08-09T14:58:54.798 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_10_[88992]: shutting down. 2024-08-09T14:58:54.799 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.799 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.799 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.799 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_13_[88995]: starting. 2024-08-09T14:58:54.799 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_13_[88995]: removed 25 objects... 2024-08-09T14:58:54.799 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_13_[88995]: removed half of the objects 2024-08-09T14:58:54.799 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_13_[88995]: removed 50 objects... 2024-08-09T14:58:54.800 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_13_[88995]: removed 50 objects 2024-08-09T14:58:54.800 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_13_[88995]: shutting down. 2024-08-09T14:58:54.800 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.800 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.800 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.800 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_11_[88993]: starting. 2024-08-09T14:58:54.801 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_11_[88993]: added 25 objects... 2024-08-09T14:58:54.801 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_11_[88993]: added half of the objects 2024-08-09T14:58:54.801 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_11_[88993]: added 50 objects... 2024-08-09T14:58:54.801 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_11_[88993]: added 50 objects 2024-08-09T14:58:54.801 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_11_[88993]: shutting down. 2024-08-09T14:58:54.801 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.802 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.802 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.802 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_12_[88994]: starting. 2024-08-09T14:58:54.802 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_12_[88994]: added 25 objects... 2024-08-09T14:58:54.802 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_12_[88994]: added half of the objects 2024-08-09T14:58:54.802 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_12_[88994]: added 50 objects... 2024-08-09T14:58:54.803 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_12_[88994]: added 50 objects 2024-08-09T14:58:54.803 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_12_[88994]: shutting down. 2024-08-09T14:58:54.803 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.803 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.803 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.803 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.803 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_14_[100867]: starting. 2024-08-09T14:58:54.803 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_14_[100867]: creating pool ceph_test_rados_list_parallel.smithi155-42268 2024-08-09T14:58:54.803 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_14_[100867]: created object 0... 2024-08-09T14:58:54.804 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_14_[100867]: created object 25... 2024-08-09T14:58:54.804 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_14_[100867]: created object 49... 2024-08-09T14:58:54.804 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_14_[100867]: finishing. 2024-08-09T14:58:54.804 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_14_[100867]: shutting down. 2024-08-09T14:58:54.804 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.804 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.804 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.804 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:58:54.804 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_15_[100869]: starting. 2024-08-09T14:58:54.804 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_15_[100869]: listing objects. 2024-08-09T14:58:54.804 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_15_[100869]: listed object 0... 2024-08-09T14:58:56.000 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_15_[100869]: list snapshots: Running main() from gmock_main.cc 2024-08-09T14:58:56.000 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-08-09T14:58:56.000 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [----------] Global test environment set-up. 2024-08-09T14:58:56.000 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-08-09T14:58:56.001 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-08-09T14:58:56.001 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (22844 ms) 2024-08-09T14:58:56.001 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-08-09T14:58:56.001 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (25825 ms) 2024-08-09T14:58:56.001 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-08-09T14:58:56.001 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (63297 ms) 2024-08-09T14:58:56.001 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-08-09T14:58:56.001 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (18018 ms) 2024-08-09T14:58:56.001 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-08-09T14:58:56.001 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (61916 ms) 2024-08-09T14:58:56.001 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (191900 ms total) 2024-08-09T14:58:56.001 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: 2024-08-09T14:58:56.002 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-08-09T14:58:56.002 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-08-09T14:58:56.002 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (33131 ms) 2024-08-09T14:58:56.002 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-08-09T14:58:56.002 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (57050 ms) 2024-08-09T14:58:56.002 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-08-09T14:58:56.002 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (94642 ms) 2024-08-09T14:58:56.002 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-08-09T14:58:56.002 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (13101 ms) 2024-08-09T14:58:56.002 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-08-09T14:58:56.002 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (14692 ms) 2024-08-09T14:58:56.002 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-08-09T14:58:56.002 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi155-42649-11. 2024-08-09T14:58:56.003 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: Waiting for snaps to purge. 2024-08-09T14:58:56.003 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (23772 ms) 2024-08-09T14:58:56.003 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (236388 ms total) 2024-08-09T14:58:56.003 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: 2024-08-09T14:58:56.003 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [----------] Global test environment tear-down 2024-08-09T14:58:56.003 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (428288 ms total) 2024-08-09T14:58:56.003 INFO:tasks.workunit.client.0.smithi155.stdout: snapshots: [ PASSED ] 11 tests. 2024-08-09T14:58:58.677 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~168s 2024-08-09T14:59:03.143 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-08-09T14:59:03.143 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-08-09T14:59:03.143 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-08-09T14:59:03.143 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-08-09T14:59:03.143 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: seed 41849 2024-08-09T14:59:03.144 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-08-09T14:59:03.144 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (32627 ms) 2024-08-09T14:59:03.144 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-08-09T14:59:03.144 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (16064 ms) 2024-08-09T14:59:03.144 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-08-09T14:59:03.144 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (81056 ms) 2024-08-09T14:59:03.144 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-08-09T14:59:03.144 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (3477 ms) 2024-08-09T14:59:03.144 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-08-09T14:59:03.144 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (14800 ms) 2024-08-09T14:59:03.144 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (148024 ms total) 2024-08-09T14:59:03.144 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: 2024-08-09T14:59:03.145 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-08-09T14:59:03.145 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-08-09T14:59:03.145 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (11357 ms) 2024-08-09T14:59:03.145 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-08-09T14:59:03.145 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (14627 ms) 2024-08-09T14:59:03.145 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-08-09T14:59:03.145 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (141539 ms) 2024-08-09T14:59:03.145 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-08-09T14:59:03.145 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (51736 ms) 2024-08-09T14:59:03.145 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-08-09T14:59:03.145 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (17630 ms) 2024-08-09T14:59:03.145 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-08-09T14:59:03.146 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.1.0-1345-g0a88632b/rpm/el9/BUILD/ceph-19.1.0-1345-g0a88632b/src/test/librados/snapshots_cxx.cc:460: Skipped 2024-08-09T14:59:03.146 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: 2024-08-09T14:59:03.146 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-08-09T14:59:03.146 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-08-09T14:59:03.146 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi155-41849-7 2024-08-09T14:59:03.146 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: waiting for snaps to purge 2024-08-09T14:59:03.146 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (23584 ms) 2024-08-09T14:59:03.146 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (260473 ms total) 2024-08-09T14:59:03.146 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: 2024-08-09T14:59:03.146 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-08-09T14:59:03.146 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-08-09T14:59:03.146 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (8 ms) 2024-08-09T14:59:03.147 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-08-09T14:59:03.147 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5762 ms) 2024-08-09T14:59:03.147 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5770 ms total) 2024-08-09T14:59:03.147 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: 2024-08-09T14:59:03.147 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-08-09T14:59:03.147 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-08-09T14:59:03.147 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (6196 ms) 2024-08-09T14:59:03.147 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-08-09T14:59:04.887 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~174s 2024-08-09T14:59:07.815 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:43.058 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:07.901 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:43.145 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:07.974 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:43.221 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:07.979 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:43.225 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:08.053 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:09:43.300 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:08.071 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:43.314 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:08.072 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:43.316 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:08.113 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:09:43.356 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:08.151 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:09:43.395 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:11.115 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~180s 2024-08-09T14:59:17.325 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~186s 2024-08-09T14:59:17.447 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK read_operations: Running main() from gmock_main.cc 2024-08-09T14:59:17.447 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-08-09T14:59:17.447 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [----------] Global test environment set-up. 2024-08-09T14:59:17.447 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-08-09T14:59:17.447 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-08-09T14:59:17.447 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (36032 ms) 2024-08-09T14:59:17.447 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-08-09T14:59:17.447 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (57427 ms) 2024-08-09T14:59:17.447 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-08-09T14:59:17.447 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (17405 ms) 2024-08-09T14:59:17.447 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-08-09T14:59:17.448 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (88419 ms) 2024-08-09T14:59:17.448 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-08-09T14:59:17.448 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (44787 ms) 2024-08-09T14:59:17.448 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-08-09T14:59:17.448 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (18866 ms) 2024-08-09T14:59:17.448 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-08-09T14:59:17.448 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (46700 ms) 2024-08-09T14:59:17.448 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-08-09T14:59:17.448 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (32124 ms) 2024-08-09T14:59:17.448 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-08-09T14:59:17.448 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (34951 ms) 2024-08-09T14:59:17.448 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-08-09T14:59:17.448 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (11038 ms) 2024-08-09T14:59:17.448 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-08-09T14:59:17.449 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (32701 ms) 2024-08-09T14:59:17.449 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-08-09T14:59:17.449 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (12824 ms) 2024-08-09T14:59:17.449 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-08-09T14:59:17.449 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (6827 ms) 2024-08-09T14:59:17.449 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-08-09T14:59:17.449 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (9642 ms) 2024-08-09T14:59:17.449 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (449743 ms total) 2024-08-09T14:59:17.449 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: 2024-08-09T14:59:17.449 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [----------] Global test environment tear-down 2024-08-09T14:59:17.449 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (449743 ms total) 2024-08-09T14:59:17.449 INFO:tasks.workunit.client.0.smithi155.stdout: read_operations: [ PASSED ] 14 tests. 2024-08-09T14:59:18.987 INFO:tasks.workunit.client.0.smithi155.stdout:ed object 25... 2024-08-09T14:59:18.987 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_15_[100869]: listed object 50... 2024-08-09T14:59:18.987 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_15_[100869]: listed object 75... 2024-08-09T14:59:18.988 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_15_[100869]: listed object 100... 2024-08-09T14:59:18.988 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_15_[100869]: listed object 125... 2024-08-09T14:59:18.988 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_15_[100869]: saw 150 objects 2024-08-09T14:59:18.988 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_15_[100869]: shutting down. 2024-08-09T14:59:18.988 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:59:18.988 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:59:18.988 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:59:18.988 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:59:18.988 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_16_[100872]: starting. 2024-08-09T14:59:18.988 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_16_[100872]: added 25 objects... 2024-08-09T14:59:18.989 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_16_[100872]: added half of the objects 2024-08-09T14:59:18.989 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_16_[100872]: added 50 objects... 2024-08-09T14:59:18.989 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_16_[100872]: added 50 objects 2024-08-09T14:59:18.989 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: process_16_[100872]: shutting down. 2024-08-09T14:59:18.989 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:59:18.989 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:59:18.989 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:59:18.989 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:59:18.989 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******************************* 2024-08-09T14:59:18.989 INFO:tasks.workunit.client.0.smithi155.stdout: list_parallel: ******* SUCCESS ********** 2024-08-09T14:59:20.119 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: Running main() from gmock_main.cc 2024-08-09T14:59:20.119 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-08-09T14:59:20.119 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [----------] Global test environment set-up. 2024-08-09T14:59:20.119 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-08-09T14:59:20.119 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-08-09T14:59:20.119 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.TooBig (4505 ms) 2024-08-09T14:59:20.119 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-08-09T14:59:20.120 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (34030 ms) 2024-08-09T14:59:20.120 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-08-09T14:59:20.120 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (14719 ms) 2024-08-09T14:59:20.120 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-08-09T14:59:20.120 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (20235 ms) 2024-08-09T14:59:20.120 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-08-09T14:59:20.120 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (20547 ms) 2024-08-09T14:59:20.120 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-08-09T14:59:20.120 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (15206 ms) 2024-08-09T14:59:20.120 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-08-09T14:59:20.121 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (55633 ms) 2024-08-09T14:59:20.121 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-08-09T14:59:20.121 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (20547 ms) 2024-08-09T14:59:20.121 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-08-09T14:59:20.121 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (29028 ms) 2024-08-09T14:59:20.121 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-08-09T14:59:20.121 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (15627 ms) 2024-08-09T14:59:20.122 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-08-09T14:59:20.122 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (47828 ms) 2024-08-09T14:59:20.122 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-08-09T14:59:20.122 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (9135 ms) 2024-08-09T14:59:20.122 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-08-09T14:59:20.122 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (14068 ms) 2024-08-09T14:59:20.122 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-08-09T14:59:20.122 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (22616 ms) 2024-08-09T14:59:20.123 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-08-09T14:59:20.123 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.Flush (40621 ms) 2024-08-09T14:59:20.123 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-08-09T14:59:20.123 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (18265 ms) 2024-08-09T14:59:20.123 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-08-09T14:59:20.123 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (7707 ms) 2024-08-09T14:59:20.123 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-08-09T14:59:20.123 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (7106 ms) 2024-08-09T14:59:20.123 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-08-09T14:59:20.123 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (6558 ms) 2024-08-09T14:59:20.124 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-08-09T14:59:20.124 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (6567 ms) 2024-08-09T14:59:20.124 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-08-09T14:59:20.124 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (8133 ms) 2024-08-09T14:59:20.124 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-08-09T14:59:20.124 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (6817 ms) 2024-08-09T14:59:20.124 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-08-09T14:59:20.124 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (7241 ms) 2024-08-09T14:59:20.125 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-08-09T14:59:20.125 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (6328 ms) 2024-08-09T14:59:20.125 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-08-09T14:59:20.125 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (6956 ms) 2024-08-09T14:59:20.125 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-08-09T14:59:20.125 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (6915 ms) 2024-08-09T14:59:20.125 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [----------] 26 tests from LibRadosAio (452938 ms total) 2024-08-09T14:59:20.125 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: 2024-08-09T14:59:20.125 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-08-09T14:59:23.535 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~193s 2024-08-09T14:59:24.271 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAioE open_pools_parallel: process_1_[42349]: starting. 2024-08-09T14:59:24.271 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_1_[42349]: creating pool ceph_test_rados_open_pools_parallel.smithi155-42309 2024-08-09T14:59:24.271 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_1_[42349]: created object 0... 2024-08-09T14:59:24.271 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_1_[42349]: created object 25... 2024-08-09T14:59:24.271 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_1_[42349]: created object 49... 2024-08-09T14:59:24.272 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_1_[42349]: finishing. 2024-08-09T14:59:24.272 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_1_[42349]: shutting down. 2024-08-09T14:59:24.272 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_2_[42350]: starting. 2024-08-09T14:59:24.272 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_2_[42350]: rados_pool_create. 2024-08-09T14:59:24.272 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_2_[42350]: rados_ioctx_create. 2024-08-09T14:59:24.272 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_2_[42350]: shutting down. 2024-08-09T14:59:24.272 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: ******************************* 2024-08-09T14:59:24.272 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_3_[87932]: starting. 2024-08-09T14:59:24.272 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_3_[87932]: creating pool ceph_test_rados_open_pools_parallel.smithi155-42309 2024-08-09T14:59:24.272 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_3_[87932]: created object 0... 2024-08-09T14:59:24.272 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_3_[87932]: created object 25... 2024-08-09T14:59:24.272 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_3_[87932]: created object 49... 2024-08-09T14:59:24.272 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_3_[87932]: finishing. 2024-08-09T14:59:24.273 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_3_[87932]: shutting down. 2024-08-09T14:59:24.273 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: ******************************* 2024-08-09T14:59:24.273 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_4_[87933]: starting. 2024-08-09T14:59:24.273 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_4_[87933]: rados_pool_create. 2024-08-09T14:59:24.273 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_4_[87933]: rados_ioctx_create. 2024-08-09T14:59:24.273 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: process_4_[87933]: shutting down. 2024-08-09T14:59:24.273 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: ******************************* 2024-08-09T14:59:24.273 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: ******************************* 2024-08-09T14:59:24.273 INFO:tasks.workunit.client.0.smithi155.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-08-09T14:59:28.223 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:03.466 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:28.226 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:03.469 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:28.232 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:03.478 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:28.236 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:03.483 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:28.296 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:03.540 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:28.301 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:03.545 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:28.326 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:03.570 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:28.327 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:03.573 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:28.411 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:03.654 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:29.746 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~199s 2024-08-09T14:59:31.545 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: Running main() from gmock_main.cc 2024-08-09T14:59:31.545 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-08-09T14:59:31.545 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [----------] Global test environment set-up. 2024-08-09T14:59:31.545 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-08-09T14:59:31.545 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-08-09T14:59:31.545 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (32907 ms) 2024-08-09T14:59:31.545 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-08-09T14:59:31.546 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (17681 ms) 2024-08-09T14:59:31.546 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-08-09T14:59:31.546 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (28917 ms) 2024-08-09T14:59:31.546 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-08-09T14:59:31.546 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (5953 ms) 2024-08-09T14:59:31.546 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (85458 ms total) 2024-08-09T14:59:31.546 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: 2024-08-09T14:59:31.546 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-08-09T14:59:31.546 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-08-09T14:59:31.546 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (241532 ms) 2024-08-09T14:59:31.546 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-08-09T14:59:31.546 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (43813 ms) 2024-08-09T14:59:31.546 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-08-09T14:59:31.547 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (23855 ms) 2024-08-09T14:59:31.547 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (309200 ms total) 2024-08-09T14:59:31.547 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: 2024-08-09T14:59:31.547 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-08-09T14:59:31.547 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-08-09T14:59:31.547 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (15993 ms) 2024-08-09T14:59:31.547 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-08-09T14:59:31.547 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (4180 ms) 2024-08-09T14:59:31.547 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-08-09T14:59:31.547 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (9451 ms) 2024-08-09T14:59:31.547 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-08-09T14:59:31.547 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (3127 ms) 2024-08-09T14:59:31.548 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (32751 ms total) 2024-08-09T14:59:31.548 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: 2024-08-09T14:59:31.548 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-08-09T14:59:31.548 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-08-09T14:59:31.548 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (10822 ms) 2024-08-09T14:59:31.548 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-08-09T14:59:31.548 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (5631 ms) 2024-08-09T14:59:31.548 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (16453 ms total) 2024-08-09T14:59:31.548 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: 2024-08-09T14:59:31.548 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [----------] Global test environment tear-down 2024-08-09T14:59:31.548 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (464235 ms total) 2024-08-09T14:59:31.548 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-08-09T14:59:34.093 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:09.337 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:34.150 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:09.393 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:34.166 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:09.413 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:34.171 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:09.418 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:34.183 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:09.427 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:34.211 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:09.458 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:34.223 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:09.466 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:34.228 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:09.471 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:34.259 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:09.502 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:35.957 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~205s 2024-08-09T14:59:42.168 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~211s 2024-08-09T14:59:43.556 INFO:tasks.workunit.client.0.smithi155.stdout: ] LibRadosSnapshotsECPP.SnapRemovePP (3230 ms) 2024-08-09T14:59:43.556 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-08-09T14:59:43.556 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (6903 ms) 2024-08-09T14:59:43.557 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-08-09T14:59:43.557 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (3134 ms) 2024-08-09T14:59:43.557 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (19464 ms total) 2024-08-09T14:59:43.557 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: 2024-08-09T14:59:43.557 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-08-09T14:59:43.557 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-08-09T14:59:43.557 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (9367 ms) 2024-08-09T14:59:43.557 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-08-09T14:59:43.557 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (7223 ms) 2024-08-09T14:59:43.557 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-08-09T14:59:43.557 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4458 ms) 2024-08-09T14:59:43.557 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (21049 ms total) 2024-08-09T14:59:43.558 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: 2024-08-09T14:59:43.558 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-08-09T14:59:43.558 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (476209 ms total) 2024-08-09T14:59:43.558 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-08-09T14:59:43.558 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-08-09T14:59:43.558 INFO:tasks.workunit.client.0.smithi155.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-08-09T14:59:48.379 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~217s 2024-08-09T14:59:48.424 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:23.668 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:48.433 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:23.676 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:48.446 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:23.692 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:48.456 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:23.703 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:48.504 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:23.747 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:48.505 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:23.749 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:48.514 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:23.757 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:48.536 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:23.783 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:48.570 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:23.813 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T14:59:50.957 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-08-09T14:59:50.957 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-08-09T14:59:50.957 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-08-09T14:59:50.958 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-08-09T14:59:50.958 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-08-09T14:59:50.958 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-08-09T14:59:50.958 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-08-09T14:59:50.958 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: 2024-08-09T14:59:50.958 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-08-09T14:59:50.958 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-08-09T14:59:50.959 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (31538 ms) 2024-08-09T14:59:50.959 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-08-09T14:59:50.959 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (102599 ms) 2024-08-09T14:59:50.959 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-08-09T14:59:50.959 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (13031 ms) 2024-08-09T14:59:50.959 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-08-09T14:59:50.959 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (39555 ms) 2024-08-09T14:59:50.959 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-08-09T14:59:50.959 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (9134 ms) 2024-08-09T14:59:50.959 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-08-09T14:59:50.960 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (37952 ms) 2024-08-09T14:59:50.960 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-08-09T14:59:50.960 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (249673 ms) 2024-08-09T14:59:50.960 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (483482 ms total) 2024-08-09T14:59:50.960 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: 2024-08-09T14:59:50.960 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-08-09T14:59:50.960 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (483482 ms total) 2024-08-09T14:59:50.960 INFO:tasks.workunit.client.0.smithi155.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-08-09T14:59:54.588 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~224s 2024-08-09T15:00:00.799 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~230s 2024-08-09T15:00:07.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~236s 2024-08-09T15:00:08.388 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:43.634 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:08.389 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:43.636 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:08.401 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:43.644 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:08.404 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:43.648 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:08.454 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:10:43.701 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:08.475 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:43.718 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:08.486 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:43.730 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:08.509 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:10:43.753 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:08.560 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:10:43.803 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:09.772 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-08-09T15:00:09.772 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-08-09T15:00:09.772 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-08-09T15:00:09.772 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-08-09T15:00:09.773 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-08-09T15:00:09.773 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (7459 ms) 2024-08-09T15:00:09.773 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-08-09T15:00:09.773 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (329578 ms) 2024-08-09T15:00:09.773 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-08-09T15:00:09.773 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (45570 ms) 2024-08-09T15:00:09.773 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-08-09T15:00:09.773 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (7746 ms) 2024-08-09T15:00:09.773 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-08-09T15:00:09.773 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (9304 ms) 2024-08-09T15:00:09.773 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-08-09T15:00:09.773 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (6693 ms) 2024-08-09T15:00:09.773 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-08-09T15:00:09.774 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (6360 ms) 2024-08-09T15:00:09.774 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-08-09T15:00:09.774 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (4884 ms) 2024-08-09T15:00:09.774 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-08-09T15:00:09.774 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (5724 ms) 2024-08-09T15:00:09.774 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-08-09T15:00:09.774 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (5492 ms) 2024-08-09T15:00:09.774 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-08-09T15:00:09.774 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (4980 ms) 2024-08-09T15:00:09.774 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-08-09T15:00:09.774 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (4260 ms) 2024-08-09T15:00:09.775 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-08-09T15:00:09.775 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (5827 ms) 2024-08-09T15:00:09.775 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-08-09T15:00:09.775 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (4236 ms) 2024-08-09T15:00:09.775 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-08-09T15:00:09.775 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (5818 ms) 2024-08-09T15:00:09.775 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-08-09T15:00:09.775 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (5201 ms) 2024-08-09T15:00:09.775 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-08-09T15:00:09.775 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3108 ms) 2024-08-09T15:00:09.775 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-08-09T15:00:09.775 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (4112 ms) 2024-08-09T15:00:09.775 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-08-09T15:00:09.776 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (4318 ms) 2024-08-09T15:00:09.776 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-08-09T15:00:09.776 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3481 ms) 2024-08-09T15:00:09.776 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-08-09T15:00:09.776 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (4328 ms) 2024-08-09T15:00:09.776 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-08-09T15:00:09.776 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (4058 ms) 2024-08-09T15:00:09.776 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-08-09T15:00:09.776 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (4296 ms) 2024-08-09T15:00:09.776 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-08-09T15:00:09.776 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (4362 ms) 2024-08-09T15:00:09.776 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-08-09T15:00:09.777 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3621 ms) 2024-08-09T15:00:09.777 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-08-09T15:00:09.777 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3443 ms) 2024-08-09T15:00:09.777 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-08-09T15:00:13.220 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~242s 2024-08-09T15:00:14.430 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio_pp: [ OK ] LibRadosA api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-08-09T15:00:14.430 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-08-09T15:00:14.430 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-08-09T15:00:14.430 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: seed 41701 2024-08-09T15:00:14.430 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-08-09T15:00:14.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (89794 ms) 2024-08-09T15:00:14.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-08-09T15:00:14.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (44605 ms) 2024-08-09T15:00:14.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-08-09T15:00:14.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (24644 ms) 2024-08-09T15:00:14.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (159043 ms total) 2024-08-09T15:00:14.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: 2024-08-09T15:00:14.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-08-09T15:00:14.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-08-09T15:00:14.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (187924 ms) 2024-08-09T15:00:14.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-08-09T15:00:14.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (92063 ms) 2024-08-09T15:00:14.431 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-08-09T15:00:14.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 19 more times... 2024-08-09T15:00:14.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing bar in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT on 2 objects, waiting a bit for snap trimming before retrying 18 more times... 2024-08-09T15:00:14.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing bar in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT on 2 objects, waiting a bit for snap trimming before retrying 17 more times... 2024-08-09T15:00:14.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing bar in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT on 2 objects, waiting a bit for snap trimming before retrying 16 more times... 2024-08-09T15:00:14.432 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing bar in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT on 2 objects, waiting a bit for snap trimming before retrying 15 more times... 2024-08-09T15:00:14.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing bar in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT on 2 objects, waiting a bit for snap trimming before retrying 14 more times... 2024-08-09T15:00:14.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing bar in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT on 2 objects, waiting a bit for snap trimming before retrying 13 more times... 2024-08-09T15:00:14.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing bar in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT on 2 objects, waiting a bit for snap trimming before retrying 12 more times... 2024-08-09T15:00:14.433 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing bar in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT on 2 objects, waiting a bit for snap trimming before retrying 11 more times... 2024-08-09T15:00:14.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:14.434 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT removing bar in ns test-rados-api-smithi155-41701-12 2024-08-09T15:00:18.658 INFO:tasks.workunit.client.0.smithi155.stdout: api_tier_pp: got ENOENT on 2 objects, write_operations: Running main() from gmock_main.cc 2024-08-09T15:00:18.658 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-08-09T15:00:18.658 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [----------] Global test environment set-up. 2024-08-09T15:00:18.658 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-08-09T15:00:18.658 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-08-09T15:00:18.658 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (36594 ms) 2024-08-09T15:00:18.659 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-08-09T15:00:18.659 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (31523 ms) 2024-08-09T15:00:18.659 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-08-09T15:00:18.659 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (24229 ms) 2024-08-09T15:00:18.659 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-08-09T15:00:18.659 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (125970 ms) 2024-08-09T15:00:18.659 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-08-09T15:00:18.659 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (9009 ms) 2024-08-09T15:00:18.659 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-08-09T15:00:18.659 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (26257 ms) 2024-08-09T15:00:18.659 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-08-09T15:00:18.659 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (257348 ms) 2024-08-09T15:00:18.660 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (510930 ms total) 2024-08-09T15:00:18.660 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: 2024-08-09T15:00:18.660 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [----------] Global test environment tear-down 2024-08-09T15:00:18.660 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (510930 ms total) 2024-08-09T15:00:18.660 INFO:tasks.workunit.client.0.smithi155.stdout: write_operations: [ PASSED ] 7 tests. 2024-08-09T15:00:19.430 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~248s 2024-08-09T15:00:25.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~255s 2024-08-09T15:00:28.520 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:03.763 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:28.523 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:03.767 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:28.579 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:03.826 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:28.580 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:03.823 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:28.580 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:03.827 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:28.581 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:03.825 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:28.582 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:03.826 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:28.650 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:03.894 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:28.658 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:03.904 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:31.851 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~261s 2024-08-09T15:00:34.869 INFO:tasks.workunit.client.0.smithi155.stdout: list: Running main() from gmock_main.cc 2024-08-09T15:00:34.869 INFO:tasks.workunit.client.0.smithi155.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-08-09T15:00:34.869 INFO:tasks.workunit.client.0.smithi155.stdout: list: [----------] Global test environment set-up. 2024-08-09T15:00:34.869 INFO:tasks.workunit.client.0.smithi155.stdout: list: [----------] 3 tests from NeoradosList 2024-08-09T15:00:34.869 INFO:tasks.workunit.client.0.smithi155.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-08-09T15:00:34.870 INFO:tasks.workunit.client.0.smithi155.stdout: list: [ OK ] NeoradosList.ListObjects (127540 ms) 2024-08-09T15:00:34.870 INFO:tasks.workunit.client.0.smithi155.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-08-09T15:00:34.870 INFO:tasks.workunit.client.0.smithi155.stdout: list: [ OK ] NeoradosList.ListObjectsNS (210314 ms) 2024-08-09T15:00:34.870 INFO:tasks.workunit.client.0.smithi155.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-08-09T15:00:34.870 INFO:tasks.workunit.client.0.smithi155.stdout: list: [ OK ] NeoradosList.ListObjectsMany (189366 ms) 2024-08-09T15:00:34.870 INFO:tasks.workunit.client.0.smithi155.stdout: list: [----------] 3 tests from NeoradosList (527220 ms total) 2024-08-09T15:00:34.870 INFO:tasks.workunit.client.0.smithi155.stdout: list: 2024-08-09T15:00:34.870 INFO:tasks.workunit.client.0.smithi155.stdout: list: [----------] Global test environment tear-down 2024-08-09T15:00:34.870 INFO:tasks.workunit.client.0.smithi155.stdout: list: [==========] 3 tests from 1 test suite ran. (527220 ms total) 2024-08-09T15:00:34.870 INFO:tasks.workunit.client.0.smithi155.stdout: list: [ PASSED ] 3 tests. 2024-08-09T15:00:38.061 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~267s 2024-08-09T15:00:44.270 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~273s 2024-08-09T15:00:48.821 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:24.064 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:48.833 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:24.076 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:48.890 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:24.133 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:48.897 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:24.140 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:48.897 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:24.144 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:48.904 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:24.151 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:48.905 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:24.148 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:48.962 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:24.206 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:48.969 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:24.216 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T15:00:50.479 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~279s 2024-08-09T15:00:56.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~286s 2024-08-09T15:01:02.898 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~292s 2024-08-09T15:01:08.668 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:43.912 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T15:01:08.721 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:43.967 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T15:01:08.722 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:43.969 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T15:01:08.734 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:43.978 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T15:01:08.776 INFO:tasks.ceph.mon.b.smithi194.stderr:==00:00:11:44.022 35135== Warning: unimplemented fcntl command: 1036 2024-08-09T15:01:08.783 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:44.026 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T15:01:08.785 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:44.029 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T15:01:08.808 INFO:tasks.ceph.mon.a.smithi155.stderr:==00:00:11:44.052 35667== Warning: unimplemented fcntl command: 1036 2024-08-09T15:01:08.849 INFO:tasks.ceph.mon.c.smithi155.stderr:==00:00:11:44.092 35672== Warning: unimplemented fcntl command: 1036 2024-08-09T15:01:09.107 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~298s 2024-08-09T15:01:15.315 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~304s 2024-08-09T15:01:15.316 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2024-08-09T15:01:16.524 INFO:tasks.ceph.osd.0:Sent signal 15 2024-08-09T15:01:16.525 INFO:tasks.ceph.osd.1:Sent signal 15 2024-08-09T15:01:16.525 INFO:tasks.ceph.osd.2:Sent signal 15 2024-08-09T15:01:16.525 INFO:tasks.ceph.osd.4:Sent signal 15 2024-08-09T15:01:16.525 INFO:tasks.ceph.osd.5:Sent signal 15 2024-08-09T15:01:16.526 INFO:tasks.ceph.osd.6:Sent signal 15 2024-08-09T15:01:16.526 INFO:tasks.ceph.osd.7:Sent signal 15 2024-08-09T15:01:16.526 INFO:tasks.ceph.mon.a:Sent signal 15 2024-08-09T15:01:16.526 INFO:tasks.ceph.mon.c:Sent signal 15 2024-08-09T15:01:16.527 INFO:tasks.ceph.mon.b:Sent signal 15 2024-08-09T15:01:16.527 INFO:tasks.ceph.mgr.y:Sent signal 15 2024-08-09T15:01:16.527 INFO:tasks.ceph.mgr.x:Sent signal 15 2024-08-09T15:01:16.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T15:01:16.526+0000 7f2c86fe0640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 35679) UID: 0 2024-08-09T15:01:16.528 INFO:tasks.ceph.mgr.y.smithi155.stderr:2024-08-09T15:01:16.526+0000 7f2c86fe0640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-08-09T15:01:16.537 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T15:01:16.528+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35960) UID: 0 2024-08-09T15:01:16.538 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T15:01:16.529+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 35673) UID: 0 2024-08-09T15:01:16.539 INFO:tasks.ceph.osd.4.smithi194.stderr:2024-08-09T15:01:16.530+0000 b485640 -1 osd.4 488 *** Got signal Terminated *** 2024-08-09T15:01:16.540 INFO:tasks.ceph.osd.0.smithi155.stderr:2024-08-09T15:01:16.529+0000 b485640 -1 osd.0 488 *** Got signal Terminated *** 2024-08-09T15:01:16.542 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T15:01:16.529+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35966) UID: 0 2024-08-09T15:01:16.543 INFO:tasks.ceph.osd.1.smithi155.stderr:2024-08-09T15:01:16.531+0000 b485640 -1 osd.1 488 *** Got signal Terminated *** 2024-08-09T15:01:16.543 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T15:01:16.529+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35973) UID: 0 2024-08-09T15:01:16.543 INFO:tasks.ceph.osd.2.smithi155.stderr:2024-08-09T15:01:16.531+0000 b485640 -1 osd.2 488 *** Got signal Terminated *** 2024-08-09T15:01:16.548 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-08-09T15:01:16.531+0000 9fbd640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35642) UID: 0 2024-08-09T15:01:16.548 INFO:tasks.ceph.mon.c.smithi155.stderr:2024-08-09T15:01:16.532+0000 9fbd640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-08-09T15:01:16.552 INFO:tasks.ceph.mon.a.smithi155.stderr:2024-08-09T15:01:16.531+0000 9fbd640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35640) UID: 0 2024-08-09T15:01:16.552 INFO:tasks.ceph.mon.a.smithi155.stderr:2024-08-09T15:01:16.534+0000 9fbd640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-08-09T15:01:16.553 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T15:01:16.542+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 35676) UID: 0 2024-08-09T15:01:16.553 INFO:tasks.ceph.osd.5.smithi194.stderr:2024-08-09T15:01:16.544+0000 b485640 -1 osd.5 488 *** Got signal Terminated *** 2024-08-09T15:01:16.556 INFO:tasks.ceph.osd.6.smithi194.stderr:2024-08-09T15:01:16.542+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 35678) UID: 0 2024-08-09T15:01:16.562 INFO:tasks.ceph.mon.b.smithi194.stderr:2024-08-09T15:01:16.542+0000 9fbd640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35130) UID: 0 2024-08-09T15:01:16.562 INFO:tasks.ceph.osd.6.smithi194.stderr:2024-08-09T15:01:16.544+0000 b485640 -1 osd.6 488 *** Got signal Terminated *** 2024-08-09T15:01:16.564 INFO:tasks.ceph.mon.b.smithi194.stderr:2024-08-09T15:01:16.544+0000 9fbd640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-08-09T15:01:16.739 INFO:tasks.ceph.mgr.x.smithi194.stderr:daemon-helper: command crashed with signal 15 2024-08-09T15:01:22.028 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T15:01:22.013+0000 b485640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 35681) UID: 0 2024-08-09T15:01:22.029 INFO:tasks.ceph.osd.7.smithi194.stderr:2024-08-09T15:01:22.014+0000 b485640 -1 osd.7 480 *** Got signal Terminated *** 2024-08-09T15:41:11.401 INFO:tasks.workunit.client.0.smithi155.stdout:C.SimpleWrite 2024-08-09T15:41:11.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (8249 ms) 2024-08-09T15:41:11.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-08-09T15:41:11.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7488 ms) 2024-08-09T15:41:11.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-08-09T15:41:11.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7692 ms) 2024-08-09T15:41:11.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-08-09T15:41:11.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (9569 ms) 2024-08-09T15:41:11.402 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-08-09T15:41:11.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7743 ms) 2024-08-09T15:41:11.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-08-09T15:41:11.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (8133 ms) 2024-08-09T15:41:11.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-08-09T15:41:11.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7572 ms) 2024-08-09T15:41:11.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-08-09T15:41:11.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (8007 ms) 2024-08-09T15:41:11.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-08-09T15:41:11.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (10280 ms) 2024-08-09T15:41:11.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-08-09T15:41:11.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (8331 ms) 2024-08-09T15:41:11.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-08-09T15:41:11.403 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (9770 ms) 2024-08-09T15:41:11.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-08-09T15:41:11.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6953 ms) 2024-08-09T15:41:11.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-08-09T15:41:11.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7758 ms) 2024-08-09T15:41:11.404 INFO:tasks.workunit.client.0.smithi155.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-08-09T15:41:11.404 INFO:tasks.workunit.client.0.smithi155.stderr:bash: line 1: 41460 Alarm clock ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 2024-08-09T15:41:11.404 INFO:tasks.workunit.client.0.smithi155.stderr: 41461 Done | tee ceph_test_rados_api_aio.log 2024-08-09T15:41:11.405 INFO:tasks.workunit.client.0.smithi155.stderr: 41463 Done | sed "s/^/ api_aio: /" 2024-08-09T20:51:47.142 INFO:tasks.workunit.client.0.smithi155.stderr:++ cleanup 2024-08-09T20:51:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:++ pkill -P 41386 2024-08-09T20:51:47.228 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-08-09T20:51:47.229 INFO:tasks.workunit.client.0.smithi155.stderr:++ true 2024-08-09T20:51:47.229 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-08-09T20:51:47.229 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-09T20:51:47.506 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi155 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a88632b45f25ae38493667c7dbd155f6e59b499 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2024-08-09T20:51:48.230 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e4350912ae044b9d8587cd7fce2d5015 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/workunit.py", line 126, in task with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test rados/test.sh) on smithi155 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a88632b45f25ae38493667c7dbd155f6e59b499 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh' 2024-08-09T20:51:48.234 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-08-09T20:51:48.244 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-08-09T20:51:48.245 DEBUG:teuthology.orchestra.run.smithi155:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-08-09T20:51:48.351 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.349+0000 7f0ee614b640 10 monclient: get_monmap_and_config 2024-08-09T20:51:48.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.349+0000 7f0ee614b640 10 monclient: build_initial_monmap 2024-08-09T20:51:48.352 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.349+0000 7f0ee614b640 10 monclient: monmap: 2024-08-09T20:51:48.352 INFO:teuthology.orchestra.run.smithi155.stderr:epoch 0 2024-08-09T20:51:48.352 INFO:teuthology.orchestra.run.smithi155.stderr:fsid f3d21f4d-627f-4083-bf15-ae5f3fe632de 2024-08-09T20:51:48.352 INFO:teuthology.orchestra.run.smithi155.stderr:last_changed 2024-08-09T20:51:48.350806+0000 2024-08-09T20:51:48.352 INFO:teuthology.orchestra.run.smithi155.stderr:created 2024-08-09T20:51:48.350806+0000 2024-08-09T20:51:48.352 INFO:teuthology.orchestra.run.smithi155.stderr:min_mon_release 0 (unknown) 2024-08-09T20:51:48.353 INFO:teuthology.orchestra.run.smithi155.stderr:election_strategy: 1 2024-08-09T20:51:48.353 INFO:teuthology.orchestra.run.smithi155.stderr:0: v1:172.21.15.155:6789/0 mon.noname-a-legacy 2024-08-09T20:51:48.353 INFO:teuthology.orchestra.run.smithi155.stderr:1: v1:172.21.15.194:6789/0 mon.noname-c-legacy 2024-08-09T20:51:48.353 INFO:teuthology.orchestra.run.smithi155.stderr:2: [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] mon.noname-b 2024-08-09T20:51:48.353 INFO:teuthology.orchestra.run.smithi155.stderr:3: v2:172.21.15.155:3300/0 mon.noname-a 2024-08-09T20:51:48.353 INFO:teuthology.orchestra.run.smithi155.stderr:4: v2:172.21.15.194:3300/0 mon.noname-c 2024-08-09T20:51:48.353 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T20:51:48.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.350+0000 7f0ee614b640 1 Processor -- start 2024-08-09T20:51:48.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.350+0000 7f0ee614b640 1 -- start start 2024-08-09T20:51:48.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.350+0000 7f0ee614b640 10 monclient: init 2024-08-09T20:51:48.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0ee614b640 10 monclient: _reopen_session rank -1 2024-08-09T20:51:48.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0ee614b640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-08-09T20:51:48.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0ee614b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ee00aadd0 addr v1:172.21.15.194:6789/0 2024-08-09T20:51:48.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0ee614b640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ee00ab1b0 0x7f0ee00adee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:51:48.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0ee614b640 10 monclient(hunting): picked mon.noname-a con 0x7f0ee00ab1b0 addr v2:172.21.15.155:3300/0 2024-08-09T20:51:48.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0ee614b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ee00ae420 addr v1:172.21.15.155:6789/0 2024-08-09T20:51:48.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0ee614b640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ee00ae420 2024-08-09T20:51:48.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0ee614b640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ab5d0 con 0x7f0ee00aadd0 2024-08-09T20:51:48.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0ee614b640 10 monclient(hunting): start opening mon connection 2024-08-09T20:51:48.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0ee614b640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ee00ab1b0 2024-08-09T20:51:48.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0ee614b640 10 monclient(hunting): _renew_subs 2024-08-09T20:51:48.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0ee614b640 10 monclient(hunting): authenticate will time out at 22915.388330s 2024-08-09T20:51:48.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ee00ae420 legacy=0x7f0ee0149960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:51:48.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ee00ab1b0 msgr2=0x7f0ee00adee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:51:48.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ee00ab1b0 0x7f0ee00adee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:51:48.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.351+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ee00aadd0 legacy=0x7f0ee013c3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:51:48.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.552+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ee00ae420 legacy=0x7f0ee0149960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:51:48.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.552+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ee00ab1b0 msgr2=0x7f0ee00adee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:51:48.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.552+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ee00ab1b0 0x7f0ee00adee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:51:48.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.552+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ee00aadd0 legacy=0x7f0ee013c3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:51:48.955 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.953+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ee00ae420 legacy=0x7f0ee0149960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:51:48.955 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.953+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ee00ab1b0 msgr2=0x7f0ee00adee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:51:48.955 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.953+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ee00ab1b0 0x7f0ee00adee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:51:48.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:48.953+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ee00aadd0 legacy=0x7f0ee013c3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:51:49.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:49.754+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ee00ae420 legacy=0x7f0ee0149960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:51:49.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:49.754+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ee00ab1b0 msgr2=0x7f0ee00adee0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:51:49.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:49.754+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ee00ab1b0 0x7f0ee00adee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:51:49.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:49.754+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ee00aadd0 legacy=0x7f0ee013c3a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:51:51.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:51:51.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:51:51.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:51:51.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:51:51.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ee00ab1b0 msgr2=0x7f0ee00adee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:51:51.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ee00ab1b0 0x7f0ee00adee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:51:51.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ee00aadd0 legacy=0x7f0ee013c3a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:51:51.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ee00ae420 legacy=0x7f0ee0149960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:51:51.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[4,2] 2024-08-09T20:51:51.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003690 0x7f0ec0003b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:51:51.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec0003690 addr v2:172.21.15.194:3300/0 2024-08-09T20:51:51.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0004180 0x7f0ec000a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:51:51.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0004180 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:51:51.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:51:51.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0003690 2024-08-09T20:51:51.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:51:51.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0004180 2024-08-09T20:51:51.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:51:51.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0004180 msgr2=0x7f0ec000a860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:51:51.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0004180 0x7f0ec000a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:51:51.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003690 msgr2=0x7f0ec0003b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:51:51.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.351+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003690 0x7f0ec0003b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:51:51.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.552+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0004180 msgr2=0x7f0ec000a860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:51:51.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.552+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0004180 0x7f0ec000a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:51:51.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.552+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003690 msgr2=0x7f0ec0003b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:51:51.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.552+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003690 0x7f0ec0003b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:51:51.955 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.953+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0004180 msgr2=0x7f0ec000a860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:51:51.955 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.953+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0004180 0x7f0ec000a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:51:51.955 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.953+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003690 msgr2=0x7f0ec0003b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:51:51.955 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:51.953+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003690 0x7f0ec0003b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:51:52.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:52.754+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0004180 msgr2=0x7f0ec000a860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:51:52.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:52.754+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0004180 0x7f0ec000a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:51:52.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:52.754+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003690 msgr2=0x7f0ec0003b00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:51:52.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:52.754+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003690 0x7f0ec0003b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:51:54.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:51:54.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:51:54.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:51:54.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:51:54.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0004180 msgr2=0x7f0ec000a860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:51:54.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0004180 0x7f0ec000a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:51:54.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003690 msgr2=0x7f0ec0003b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:51:54.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003690 0x7f0ec0003b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:51:54.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-08-09T20:51:54.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000b270 0x7f0ec000d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:51:54.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:51:54.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec000b270 addr v2:172.21.15.194:3300/0 2024-08-09T20:51:54.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0003b00 0x7f0ec000a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:51:54.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0003b00 addr v2:172.21.15.155:3300/0 2024-08-09T20:51:54.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:51:54.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec00067c0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:51:54.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:51:54.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0003b00 2024-08-09T20:51:54.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:51:54.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec000b270 2024-08-09T20:51:54.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:51:54.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec0007380 con 0x7f0ec00067c0 2024-08-09T20:51:54.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:51:54.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0003b00 msgr2=0x7f0ec000a860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:51:54.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:51:54.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0003b00 0x7f0ec000a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:51:54.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:51:54.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000b270 msgr2=0x7f0ec000d680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:51:54.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.352+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000b270 0x7f0ec000d680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:51:54.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.553+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0003b00 msgr2=0x7f0ec000a860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:51:54.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.553+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:51:54.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.553+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0003b00 0x7f0ec000a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:51:54.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.553+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:51:54.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.553+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000b270 msgr2=0x7f0ec000d680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:51:54.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.553+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000b270 0x7f0ec000d680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:51:54.955 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.954+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:51:54.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.954+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0003b00 msgr2=0x7f0ec000a860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:51:54.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.954+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:51:54.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.954+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0003b00 0x7f0ec000a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:51:54.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.954+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000b270 msgr2=0x7f0ec000d680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:51:54.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:54.954+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000b270 0x7f0ec000d680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:51:55.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:55.755+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:51:55.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:55.755+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0003b00 msgr2=0x7f0ec000a860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:51:55.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:55.755+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:51:55.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:55.755+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0003b00 0x7f0ec000a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:51:55.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:55.755+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000b270 msgr2=0x7f0ec000d680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:51:55.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:55.755+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000b270 0x7f0ec000d680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:51:57.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:51:57.354 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:51:57.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:51:57.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:51:57.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:51:57.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:51:57.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000b270 msgr2=0x7f0ec000d680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:51:57.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000b270 0x7f0ec000d680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:51:57.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0003b00 msgr2=0x7f0ec000a860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:51:57.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0003b00 0x7f0ec000a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:51:57.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[0,1] 2024-08-09T20:51:57.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec00076c0 addr v1:172.21.15.155:6789/0 2024-08-09T20:51:57.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec0008280 addr v1:172.21.15.194:6789/0 2024-08-09T20:51:57.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ad300 con 0x7f0ec00076c0 2024-08-09T20:51:57.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec0008280 2024-08-09T20:51:57.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:51:57.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00076c0 legacy=0x7f0ec0007ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:51:57.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.352+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0008280 legacy=0x7f0ec0014c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:51:57.554 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.553+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00076c0 legacy=0x7f0ec0007ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:51:57.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.553+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0008280 legacy=0x7f0ec0014c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:51:57.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.954+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00076c0 legacy=0x7f0ec0007ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:51:57.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:57.954+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0008280 legacy=0x7f0ec0014c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:51:58.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:58.755+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00076c0 legacy=0x7f0ec0007ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:51:58.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:51:58.755+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0008280 legacy=0x7f0ec0014c20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:00.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:00.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:00.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:00.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:00.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0008280 legacy=0x7f0ec0014c20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:00.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00076c0 legacy=0x7f0ec0007ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:00.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-08-09T20:52:00.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:00.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:52:00.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec00067c0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:52:00.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec0003b00 addr v1:172.21.15.155:6789/0 2024-08-09T20:52:00.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d680 0x7f0ec000a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:00.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec000d680 addr v2:172.21.15.194:3300/0 2024-08-09T20:52:00.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ab5d0 con 0x7f0ec0003b00 2024-08-09T20:52:00.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:00.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec000d680 2024-08-09T20:52:00.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:00.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec00067c0 2024-08-09T20:52:00.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:00.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:00.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:00.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:00.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edffff640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d680 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:00.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.352+0000 7f0edffff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d680 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:00.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.553+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:00.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.553+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:00.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.553+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:00.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.553+0000 7f0edffff640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d680 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:00.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.553+0000 7f0edffff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d680 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:00.955 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.954+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:00.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.954+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:00.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.954+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:00.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.954+0000 7f0edffff640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d680 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:00.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:00.954+0000 7f0edffff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d680 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:01.756 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:01.755+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:01.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:01.755+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:01.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:01.755+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:01.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:01.755+0000 7f0edffff640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d680 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:01.757 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:01.755+0000 7f0edffff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d680 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:03.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:03.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:03.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:03.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:03.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:03.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:03.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d680 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:03.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d680 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:03.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:03.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[1,3] 2024-08-09T20:52:03.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec000fbf0 addr v1:172.21.15.194:6789/0 2024-08-09T20:52:03.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0010710 0x7f0ec0016020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:03.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0010710 addr v2:172.21.15.155:3300/0 2024-08-09T20:52:03.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ad300 con 0x7f0ec000fbf0 2024-08-09T20:52:03.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:03.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0010710 2024-08-09T20:52:03.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:03.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0010710 msgr2=0x7f0ec0016020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:03.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0010710 0x7f0ec0016020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:03.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.353+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec000fbf0 legacy=0x7f0ec0010000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:03.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.554+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0010710 msgr2=0x7f0ec0016020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:03.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.554+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0010710 0x7f0ec0016020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:03.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.554+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec000fbf0 legacy=0x7f0ec0010000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:03.957 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.955+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0010710 msgr2=0x7f0ec0016020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:03.957 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.955+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0010710 0x7f0ec0016020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:03.957 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:03.955+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec000fbf0 legacy=0x7f0ec0010000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:04.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:04.756+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0010710 msgr2=0x7f0ec0016020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:04.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:04.756+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0010710 0x7f0ec0016020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:04.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:04.756+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec000fbf0 legacy=0x7f0ec0010000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:06.355 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:06.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:06.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:06.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:06.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0010710 msgr2=0x7f0ec0016020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:06.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0010710 0x7f0ec0016020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:06.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec000fbf0 legacy=0x7f0ec0010000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:06.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-08-09T20:52:06.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:52:06.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec00165a0 addr v1:172.21.15.194:6789/0 2024-08-09T20:52:06.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec00067c0 addr v1:172.21.15.155:6789/0 2024-08-09T20:52:06.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:06.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0003b00 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:52:06.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ab5d0 con 0x7f0ec00067c0 2024-08-09T20:52:06.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ad300 con 0x7f0ec00165a0 2024-08-09T20:52:06.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:06.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0003b00 2024-08-09T20:52:06.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:06.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:06.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:06.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:06.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.353+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00165a0 legacy=0x7f0ec0016980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:06.555 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.554+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:06.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.554+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:06.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.554+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:06.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.554+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00165a0 legacy=0x7f0ec0016980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:06.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.955+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:06.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.955+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:06.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.955+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:06.956 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:06.955+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00165a0 legacy=0x7f0ec0016980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:07.758 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:07.756+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:07.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:07.756+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:07.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:07.756+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:07.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:07.756+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00165a0 legacy=0x7f0ec0016980 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:09.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:09.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:09.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:09.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:09.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:09.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:09.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00165a0 legacy=0x7f0ec0016980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:09.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:09.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[4,3] 2024-08-09T20:52:09.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec00103e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:09.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec0010000 addr v2:172.21.15.194:3300/0 2024-08-09T20:52:09.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:09.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0015010 addr v2:172.21.15.155:3300/0 2024-08-09T20:52:09.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:09.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0015010 2024-08-09T20:52:09.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:09.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0010000 2024-08-09T20:52:09.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:09.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:09.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:09.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec00103e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:09.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.354+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec00103e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:09.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.555+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:09.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.555+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:09.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.555+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec00103e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:09.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.555+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec00103e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:09.957 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.956+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:09.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.956+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:09.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.956+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec00103e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:09.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:09.956+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec00103e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:10.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:10.757+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:10.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:10.757+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:10.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:10.757+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec00103e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:10.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:10.757+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec00103e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:12.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:12.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:12.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:12.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:12.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec00103e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:12.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec00103e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:12.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:12.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:12.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-08-09T20:52:12.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:52:12.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:12.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0015010 addr v2:172.21.15.155:3300/0 2024-08-09T20:52:12.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:12.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0003b00 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:52:12.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec00067c0 addr v1:172.21.15.194:6789/0 2024-08-09T20:52:12.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec00067c0 2024-08-09T20:52:12.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:12.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0015010 2024-08-09T20:52:12.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:12.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0003b00 2024-08-09T20:52:12.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:12.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:12.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:12.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:12.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:12.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.354+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:12.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.555+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:12.556 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.555+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:12.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.555+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:12.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.555+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:12.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.555+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:12.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.956+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:12.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.956+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:12.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.956+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:12.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.956+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:12.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:12.956+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:13.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:13.758+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:13.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:13.758+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:13.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:13.758+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:13.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:13.758+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:13.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:13.758+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:15.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:15.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:15.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:15.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:15.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:15.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:15.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:15.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:15.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:15.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[4,0] 2024-08-09T20:52:15.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d660 0x7f0ec0010000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:15.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec000d660 addr v2:172.21.15.194:3300/0 2024-08-09T20:52:15.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec0015960 addr v1:172.21.15.155:6789/0 2024-08-09T20:52:15.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ab5d0 con 0x7f0ec0015960 2024-08-09T20:52:15.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:15.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec000d660 2024-08-09T20:52:15.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:15.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:15.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d660 msgr2=0x7f0ec0010000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:15.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.354+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d660 0x7f0ec0010000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:15.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.555+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:15.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.555+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d660 msgr2=0x7f0ec0010000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:15.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.555+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d660 0x7f0ec0010000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:15.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.956+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:15.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.956+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d660 msgr2=0x7f0ec0010000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:15.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:15.956+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d660 0x7f0ec0010000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:16.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:16.757+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015dd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:16.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:16.757+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d660 msgr2=0x7f0ec0010000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:16.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:16.757+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d660 0x7f0ec0010000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:18.356 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:18.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:18.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:18.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:18.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d660 msgr2=0x7f0ec0010000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:18.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec000d660 0x7f0ec0010000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:18.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:18.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-08-09T20:52:18.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec00067c0 addr v1:172.21.15.194:6789/0 2024-08-09T20:52:18.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec0015960 addr v1:172.21.15.155:6789/0 2024-08-09T20:52:18.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:18.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:52:18.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0003b00 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:52:18.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec0015960 2024-08-09T20:52:18.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ab5d0 con 0x7f0ec00067c0 2024-08-09T20:52:18.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:18.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0003b00 2024-08-09T20:52:18.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:18.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:18.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:18.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:18.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.354+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:18.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.555+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:18.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.555+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:18.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.555+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:18.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.555+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:18.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.956+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:18.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.956+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:18.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.956+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:18.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:18.956+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:19.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:19.757+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:19.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:19.757+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:19.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:19.757+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:19.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:19.757+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:21.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:21.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:21.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:21.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:21.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:21.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0003b00 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:21.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:21.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:21.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[4,3] 2024-08-09T20:52:21.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:21.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec0010000 addr v2:172.21.15.194:3300/0 2024-08-09T20:52:21.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:21.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0015010 addr v2:172.21.15.155:3300/0 2024-08-09T20:52:21.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:21.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0015010 2024-08-09T20:52:21.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:21.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0010000 2024-08-09T20:52:21.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:21.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:21.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:21.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:21.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.354+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:21.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.555+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:21.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.555+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:21.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.555+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:21.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.555+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:21.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.956+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:21.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.956+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:21.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.956+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:21.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:21.956+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:22.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:22.757+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:22.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:22.757+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:22.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:22.757+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:22.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:22.757+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:24.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.354+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:24.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.354+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:24.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.354+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:24.357 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.354+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:24.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.354+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:24.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.354+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:24.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.354+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 msgr2=0x7f0ec0015420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:24.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.354+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015010 0x7f0ec0015420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:24.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.354+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-08-09T20:52:24.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.354+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:52:24.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.354+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec0015960 addr v1:172.21.15.155:6789/0 2024-08-09T20:52:24.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.354+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:24.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.355+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec00067c0 addr v2:172.21.15.155:3300/0 2024-08-09T20:52:24.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.355+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:24.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.355+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0015010 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:52:24.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.355+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ad300 con 0x7f0ec0015960 2024-08-09T20:52:24.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.355+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:24.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.355+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec00067c0 2024-08-09T20:52:24.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.355+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:24.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.355+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0015010 2024-08-09T20:52:24.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.355+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:24.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.355+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:24.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.355+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:24.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.355+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:24.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.355+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:24.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.355+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:24.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.556+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:24.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.556+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:24.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.556+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:24.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.556+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:24.559 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.556+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:24.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.957+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:24.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.957+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:24.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.957+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:24.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.957+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:24.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:24.957+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:25.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:25.758+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:25.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:25.758+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:25.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:25.758+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:25.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:25.758+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:25.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:25.758+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:27.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:27.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:27.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:27.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:27.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:27.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:27.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:27.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:27.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015960 legacy=0x7f0ec0015d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:27.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[4,1] 2024-08-09T20:52:27.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003b00 0x7f0ec000a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:27.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec0003b00 addr v2:172.21.15.194:3300/0 2024-08-09T20:52:27.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec0010000 addr v1:172.21.15.194:6789/0 2024-08-09T20:52:27.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec0010000 2024-08-09T20:52:27.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:27.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0003b00 2024-08-09T20:52:27.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:27.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:27.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003b00 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:27.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.355+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003b00 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:27.557 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.556+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:27.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.556+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003b00 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:27.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.556+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003b00 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:27.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.957+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003b00 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:27.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.957+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:27.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:27.957+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003b00 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:28.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:28.758+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:28.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:28.758+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003b00 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:28.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:28.758+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003b00 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:30.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:30.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:30.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:30.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:30.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003b00 msgr2=0x7f0ec000a830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:30.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0003b00 0x7f0ec000a830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:30.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:30.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-08-09T20:52:30.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:52:30.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:30.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0015010 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:52:30.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec0003b00 addr v1:172.21.15.155:6789/0 2024-08-09T20:52:30.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:30.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec00067c0 addr v2:172.21.15.194:3300/0 2024-08-09T20:52:30.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ad300 con 0x7f0ec0003b00 2024-08-09T20:52:30.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:30.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec00067c0 2024-08-09T20:52:30.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:30.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:30.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0015010 2024-08-09T20:52:30.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:30.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:30.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:30.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.355+0000 7f0edffff640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:30.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.356+0000 7f0edffff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:30.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.556+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:30.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.556+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:30.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.556+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:30.559 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.557+0000 7f0edffff640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:30.559 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.557+0000 7f0edffff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:30.958 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.957+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:30.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.957+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:30.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.957+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:30.959 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.958+0000 7f0edffff640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:30.960 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:30.958+0000 7f0edffff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:31.759 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:31.758+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:31.758+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:31.760 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:31.758+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:31.759+0000 7f0edffff640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:31.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:31.759+0000 7f0edffff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:33.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:33.358 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:33.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:33.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:33.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:33.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:33.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:33.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:33.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:33.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[1,3] 2024-08-09T20:52:33.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec0010000 addr v1:172.21.15.194:6789/0 2024-08-09T20:52:33.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:33.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0016040 addr v2:172.21.15.155:3300/0 2024-08-09T20:52:33.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec0010000 2024-08-09T20:52:33.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:33.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0016040 2024-08-09T20:52:33.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:33.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:33.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:33.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.356+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:33.559 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.557+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:33.559 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.557+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:33.559 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.557+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:33.960 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.958+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:33.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.958+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:33.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:33.958+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:34.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:34.759+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:34.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:34.759+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:34.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:34.759+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:36.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:36.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:36.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:36.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:36.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:36.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:36.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:36.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-08-09T20:52:36.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:36.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec00067c0 addr v2:172.21.15.194:3300/0 2024-08-09T20:52:36.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:36.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0015010 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:52:36.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:36.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:52:36.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0016040 addr v2:172.21.15.155:3300/0 2024-08-09T20:52:36.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:36.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0016040 2024-08-09T20:52:36.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:36.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec00067c0 2024-08-09T20:52:36.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:36.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec0007380 con 0x7f0ec0015010 2024-08-09T20:52:36.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:36.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:36.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edffff640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:36.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:36.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edffff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:36.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:36.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.356+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:36.559 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.557+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:36.559 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.557+0000 7f0edffff640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:36.559 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.557+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:36.559 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.557+0000 7f0edffff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:36.560 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.557+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:36.560 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.557+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:36.960 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.958+0000 7f0edffff640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:36.960 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.958+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:36.960 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.958+0000 7f0edffff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:36.960 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.958+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:36.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.958+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:36.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:36.958+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:37.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:37.759+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:37.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:37.759+0000 7f0edffff640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:37.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:37.759+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:37.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:37.759+0000 7f0edffff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:37.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:37.759+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:37.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:37.759+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:39.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:39.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:39.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:39.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:39.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 msgr2=0x7f0ec00153f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:39.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015010 0x7f0ec00153f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:39.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:39.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:39.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:39.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:39.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[0,1] 2024-08-09T20:52:39.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec0003b00 addr v1:172.21.15.155:6789/0 2024-08-09T20:52:39.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec0010000 addr v1:172.21.15.194:6789/0 2024-08-09T20:52:39.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ad300 con 0x7f0ec0003b00 2024-08-09T20:52:39.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec0010000 2024-08-09T20:52:39.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:39.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:39.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.356+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:39.558 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.557+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:39.559 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.557+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:39.960 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.958+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:39.960 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:39.958+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:40.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:40.759+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:40.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:40.759+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:42.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:42.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:42.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:42.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:42.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:42.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:42.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-08-09T20:52:42.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:42.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0016040 addr v2:172.21.15.155:3300/0 2024-08-09T20:52:42.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:52:42.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:42.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec00067c0 addr v2:172.21.15.194:3300/0 2024-08-09T20:52:42.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:42.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0010000 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:52:42.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:42.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0016040 2024-08-09T20:52:42.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:42.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec00067c0 2024-08-09T20:52:42.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:42.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec0007380 con 0x7f0ec0010000 2024-08-09T20:52:42.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:42.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:42.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:42.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:42.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:42.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:42.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.357+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:42.560 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.558+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:42.560 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.558+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:42.560 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.558+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:42.560 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.558+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:42.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.558+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:42.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.558+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:42.960 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.959+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:42.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.959+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:42.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.959+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:42.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.959+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:42.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.959+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:42.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:42.959+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:43.761 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:43.760+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:43.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:43.760+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:43.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:43.760+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:43.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:43.760+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:43.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:43.760+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:43.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:43.760+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:45.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:45.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:45.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:45.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:45.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:45.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:45.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 msgr2=0x7f0ec0006c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:45.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec00067c0 0x7f0ec0006c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:45.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:45.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:45.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[1,0] 2024-08-09T20:52:45.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec0003b00 addr v1:172.21.15.194:6789/0 2024-08-09T20:52:45.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec0015720 addr v1:172.21.15.155:6789/0 2024-08-09T20:52:45.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ab5d0 con 0x7f0ec0015720 2024-08-09T20:52:45.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ad300 con 0x7f0ec0003b00 2024-08-09T20:52:45.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:45.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec0017550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:45.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.357+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:45.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.558+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec0017550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:45.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.558+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:45.960 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.959+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec0017550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:45.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:45.959+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:46.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:46.760+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec0017550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:46.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:46.760+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:48.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:48.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:48.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:48.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:48.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0003b00 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:48.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec0017550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:48.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-08-09T20:52:48.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:52:48.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:48.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0016040 addr v2:172.21.15.155:3300/0 2024-08-09T20:52:48.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:48.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0010000 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:52:48.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec00067c0 addr v1:172.21.15.155:6789/0 2024-08-09T20:52:48.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec00067c0 2024-08-09T20:52:48.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:48.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0016040 2024-08-09T20:52:48.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:48.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:48.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0010000 2024-08-09T20:52:48.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:48.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:48.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:48.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:48.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.358+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:48.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.559+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:48.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.559+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:48.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.559+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:48.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.559+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:48.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.559+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:48.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.960+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:48.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.960+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:48.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.960+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:48.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.960+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:48.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:48.960+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:49.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:49.761+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:49.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:49.761+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:49.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:49.761+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:49.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:49.761+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:49.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:49.761+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:51.359 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:51.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:51.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:51.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:51.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:51.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:51.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:51.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:51.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:51.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[1,4] 2024-08-09T20:52:51.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec0003ae0 addr v1:172.21.15.194:6789/0 2024-08-09T20:52:51.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:51.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec0006c50 addr v2:172.21.15.194:3300/0 2024-08-09T20:52:51.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ab5d0 con 0x7f0ec0003ae0 2024-08-09T20:52:51.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:51.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0006c50 2024-08-09T20:52:51.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:51.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:51.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0003ae0 legacy=0x7f0ec0017550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:51.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.358+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:51.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.559+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:51.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.559+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0003ae0 legacy=0x7f0ec0017550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:51.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.559+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:51.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.960+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0003ae0 legacy=0x7f0ec0017550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:51.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.960+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:51.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:51.960+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:52.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:52.761+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0003ae0 legacy=0x7f0ec0017550 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:52.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:52.761+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:52.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:52.761+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:54.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:54.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:54.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:54.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:54.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:54.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:54.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0003ae0 legacy=0x7f0ec0017550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:54.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-08-09T20:52:54.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:54.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0016040 addr v2:172.21.15.155:3300/0 2024-08-09T20:52:54.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:52:54.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec0015c60 addr v1:172.21.15.194:6789/0 2024-08-09T20:52:54.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec00067c0 addr v1:172.21.15.155:6789/0 2024-08-09T20:52:54.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec00067c0 2024-08-09T20:52:54.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ab5d0 con 0x7f0ec0015c60 2024-08-09T20:52:54.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:54.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0016040 2024-08-09T20:52:54.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:54.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:54.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:54.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:54.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.358+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015c60 legacy=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:54.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.559+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:54.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.559+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:54.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.559+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:54.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.559+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015c60 legacy=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:54.961 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.960+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:54.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.960+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:54.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.960+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:54.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:54.960+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015c60 legacy=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:55.762 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:55.761+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:52:55.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:55.761+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:52:55.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:55.761+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:55.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:55.761+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015c60 legacy=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:52:57.360 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:52:57.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:52:57.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:52:57.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:52:57.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 msgr2=0x7f0ec000a800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:57.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0016040 0x7f0ec000a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:52:57.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015c60 legacy=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:57.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:52:57.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[2,4] 2024-08-09T20:52:57.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:57.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0006c50 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:52:57.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:52:57.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec0010000 addr v2:172.21.15.194:3300/0 2024-08-09T20:52:57.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:57.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0010000 2024-08-09T20:52:57.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:52:57.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0006c50 2024-08-09T20:52:57.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:52:57.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:57.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:57.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:57.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.359+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:52:57.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.560+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:57.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.560+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:57.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.560+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:57.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.560+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:52:57.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.961+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:57.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.961+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:57.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.961+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:57.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:57.961+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:52:58.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:58.762+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:52:58.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:58.762+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:52:58.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:58.762+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:52:58.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:52:58.762+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:00.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:00.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:00.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:00.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:00.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:00.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:00.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:00.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:00.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-08-09T20:53:00.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:53:00.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec00067c0 addr v1:172.21.15.155:6789/0 2024-08-09T20:53:00.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0007aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:00.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0006c50 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:53:00.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:00.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec0010000 addr v2:172.21.15.194:3300/0 2024-08-09T20:53:00.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ad300 con 0x7f0ec00067c0 2024-08-09T20:53:00.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:00.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:00.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0010000 2024-08-09T20:53:00.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:00.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0006c50 2024-08-09T20:53:00.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:00.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:00.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:00.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edffff640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:00.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.359+0000 7f0edffff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:00.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.560+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:00.561 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.560+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:00.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.560+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:00.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.560+0000 7f0edffff640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:00.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.560+0000 7f0edffff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:00.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.961+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:00.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.961+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:00.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.961+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:00.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.961+0000 7f0edffff640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:00.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:00.961+0000 7f0edffff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:01.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:01.762+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:01.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:01.762+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:01.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:01.762+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:01.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:01.762+0000 7f0edffff640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:01.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:01.762+0000 7f0edffff640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:03.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:03.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:03.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:03.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:03.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:03.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:03.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:03.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:03.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[1,3] 2024-08-09T20:53:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec0015720 addr v1:172.21.15.194:6789/0 2024-08-09T20:53:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 0x7f0ec000e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0015b00 addr v2:172.21.15.155:3300/0 2024-08-09T20:53:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec0015720 2024-08-09T20:53:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0015b00 2024-08-09T20:53:03.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.358+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:03.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.359+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 msgr2=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:03.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.359+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 0x7f0ec000e670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:03.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.359+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec0019560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:03.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.560+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 msgr2=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:03.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.560+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 0x7f0ec000e670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:03.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.560+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec0019560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:03.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.961+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 msgr2=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:03.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.961+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 0x7f0ec000e670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:03.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:03.961+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec0019560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:04.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:04.762+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 msgr2=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:04.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:04.762+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 0x7f0ec000e670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:04.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:04.762+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec0019560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:06.361 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:06.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:06.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:06.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:06.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 msgr2=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:06.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 0x7f0ec000e670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:06.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec0019560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:06.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-08-09T20:53:06.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 0x7f0ec000e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:06.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0015b00 addr v2:172.21.15.155:3300/0 2024-08-09T20:53:06.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:53:06.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:06.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec0010000 addr v2:172.21.15.194:3300/0 2024-08-09T20:53:06.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec00067c0 addr v1:172.21.15.155:6789/0 2024-08-09T20:53:06.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ad300 con 0x7f0ec00067c0 2024-08-09T20:53:06.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:06.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0015b00 2024-08-09T20:53:06.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:06.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0010000 2024-08-09T20:53:06.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:06.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 msgr2=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:06.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 0x7f0ec000e670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:06.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:06.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:06.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.359+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:06.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.560+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 msgr2=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:06.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.560+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:06.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.560+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 0x7f0ec000e670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:06.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.560+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:06.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.560+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:06.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.961+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 msgr2=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:06.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.961+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:06.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.961+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 0x7f0ec000e670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:06.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.961+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:06.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:06.961+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:07.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:07.762+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 msgr2=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:07.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:07.762+0000 7f0edffff640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:07.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:07.762+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 0x7f0ec000e670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:07.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:07.762+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:07.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:07.762+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:09.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:09.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:09.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:09.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:09.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:09.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:09.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 msgr2=0x7f0ec000e670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:09.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0015b00 0x7f0ec000e670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:09.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:09.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[2,1] 2024-08-09T20:53:09.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0007aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:09.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0006c50 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:53:09.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec0015720 addr v1:172.21.15.194:6789/0 2024-08-09T20:53:09.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec0015720 2024-08-09T20:53:09.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:09.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0006c50 2024-08-09T20:53:09.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:09.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:09.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:09.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.359+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec000ebb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.560+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.560+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:09.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.560+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec000ebb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:09.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.961+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:09.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.961+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:09.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:09.961+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec000ebb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:10.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:10.762+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:10.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:10.762+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:10.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:10.762+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec000ebb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:12.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:12.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:12.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:12.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:12.362 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 msgr2=0x7f0ec0007aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:12.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0006c50 0x7f0ec0007aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:12.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0015720 legacy=0x7f0ec000ebb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:12.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-08-09T20:53:12.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:53:12.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec00067c0 addr v1:172.21.15.155:6789/0 2024-08-09T20:53:12.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:12.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0006c50 addr v2:172.21.15.155:3300/0 2024-08-09T20:53:12.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec0010000 addr v1:172.21.15.194:6789/0 2024-08-09T20:53:12.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ad300 con 0x7f0ec00067c0 2024-08-09T20:53:12.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec0010000 2024-08-09T20:53:12.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:12.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0006c50 2024-08-09T20:53:12.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:12.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:12.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:12.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.359+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:12.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.360+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:12.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.560+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:12.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.560+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:12.562 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.560+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:12.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.561+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:12.962 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.961+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:12.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.961+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:12.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.961+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:12.963 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:12.962+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:13.763 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:13.762+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:13.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:13.762+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:13.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:13.762+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:13.764 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:13.763+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:15.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:15.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:15.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:15.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:15.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:15.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:15.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0010000 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:15.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:15.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[4,2] 2024-08-09T20:53:15.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0007aa0 0x7f0ec000f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:15.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec0007aa0 addr v2:172.21.15.194:3300/0 2024-08-09T20:53:15.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:15.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0015e30 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:53:15.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:15.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0007aa0 2024-08-09T20:53:15.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:15.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0015e30 2024-08-09T20:53:15.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:15.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:15.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:15.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0007aa0 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:15.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.360+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0007aa0 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:15.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.561+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:15.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.561+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:15.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.561+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0007aa0 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:15.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.561+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0007aa0 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:15.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.962+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:15.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.962+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:15.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.962+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0007aa0 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:15.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:15.962+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0007aa0 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:16.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:16.763+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:16.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:16.763+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:16.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:16.763+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0007aa0 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:16.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:16.763+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0007aa0 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:18.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:18.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:18.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:18.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:18.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:18.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:18.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0007aa0 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:18.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0007aa0 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:18.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-08-09T20:53:18.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:18.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0006c50 addr v2:172.21.15.155:3300/0 2024-08-09T20:53:18.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:53:18.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec00067c0 addr v1:172.21.15.155:6789/0 2024-08-09T20:53:18.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec0007aa0 addr v1:172.21.15.194:6789/0 2024-08-09T20:53:18.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ab5d0 con 0x7f0ec00067c0 2024-08-09T20:53:18.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ad300 con 0x7f0ec0007aa0 2024-08-09T20:53:18.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:18.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0006c50 2024-08-09T20:53:18.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:18.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:18.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:18.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:18.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.360+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:18.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.561+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:18.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.561+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:18.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.561+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:18.563 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.561+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:18.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.962+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:18.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.962+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:18.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.962+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:18.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:18.962+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:19.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:19.763+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:19.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:19.763+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:19.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:19.763+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:19.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:19.763+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:21.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:21.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:21.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:21.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:21.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:21.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:21.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:21.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:21.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[4,2] 2024-08-09T20:53:21.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:21.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec0010000 addr v2:172.21.15.194:3300/0 2024-08-09T20:53:21.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:21.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0015e30 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:53:21.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:21.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0010000 2024-08-09T20:53:21.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:21.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0015e30 2024-08-09T20:53:21.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:21.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:21.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:21.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:21.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.361+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:21.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.562+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:21.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.562+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:21.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.562+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:21.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.562+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:21.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.963+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:21.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.963+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:21.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.963+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:21.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:21.963+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:22.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:22.764+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:22.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:22.764+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:22.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:22.764+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:22.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:22.764+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:24.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:24.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:24.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:24.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:24.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:24.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:24.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:24.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:24.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-08-09T20:53:24.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:24.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0006c50 addr v2:172.21.15.155:3300/0 2024-08-09T20:53:24.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:24.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0015e30 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:53:24.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:53:24.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec0007aa0 addr v1:172.21.15.194:6789/0 2024-08-09T20:53:24.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec0007aa0 2024-08-09T20:53:24.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:24.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0006c50 2024-08-09T20:53:24.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:24.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0015e30 2024-08-09T20:53:24.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:24.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:24.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:24.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:24.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:24.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.361+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:24.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.562+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:24.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.562+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:24.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.562+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:24.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.562+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:24.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.562+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:24.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.963+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:24.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.963+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:24.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.963+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:24.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.963+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:24.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:24.963+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:25.765 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:25.764+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:25.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:25.764+0000 7f0edf7fe640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:25.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:25.764+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:25.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:25.764+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:25.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:25.764+0000 7f0edffff640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:27.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:27.363 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:27.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:27.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:27.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 msgr2=0x7f0ec001a560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:27.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0015e30 0x7f0ec001a560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:27.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec000a790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:27.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec000a790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:27.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:27.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[4,0] 2024-08-09T20:53:27.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:27.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec0010000 addr v2:172.21.15.194:3300/0 2024-08-09T20:53:27.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec00067c0 addr v1:172.21.15.155:6789/0 2024-08-09T20:53:27.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ab5d0 con 0x7f0ec00067c0 2024-08-09T20:53:27.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:27.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0010000 2024-08-09T20:53:27.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:27.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:27.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:27.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.361+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:27.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.562+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:27.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.562+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:27.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.562+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:27.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.963+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:27.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.963+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:27.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:27.963+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:28.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:28.764+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:28.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:28.764+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:28.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:28.764+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:30.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:30.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:30.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:30.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:30.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:30.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:30.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:30.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-08-09T20:53:30.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:30.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:53:30.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec0010000 addr v2:172.21.15.194:3300/0 2024-08-09T20:53:30.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec00067c0 addr v1:172.21.15.155:6789/0 2024-08-09T20:53:30.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:30.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0006c50 addr v2:172.21.15.155:3300/0 2024-08-09T20:53:30.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec00067c0 2024-08-09T20:53:30.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:30.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0006c50 2024-08-09T20:53:30.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:30.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0010000 2024-08-09T20:53:30.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:30.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edffff640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:30.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edffff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:30.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:30.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:30.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.361+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:30.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.562+0000 7f0edffff640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:30.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.562+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:30.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.562+0000 7f0edffff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:30.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.562+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:30.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.562+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:30.964 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.963+0000 7f0edffff640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:30.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.963+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:30.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.963+0000 7f0edffff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:30.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.963+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:30.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:30.963+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:31.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:31.764+0000 7f0edeffd640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:31.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:31.764+0000 7f0edffff640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:31.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:31.764+0000 7f0edffff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:31.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:31.764+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:31.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:31.764+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:33.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:33.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:33.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:33.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:33.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:33.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:33.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:33.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:33.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:33.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[2,1] 2024-08-09T20:53:33.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0007aa0 0x7f0ec0015c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:33.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec0007aa0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:53:33.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec0009790 addr v1:172.21.15.194:6789/0 2024-08-09T20:53:33.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ab5d0 con 0x7f0ec0009790 2024-08-09T20:53:33.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:33.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0007aa0 2024-08-09T20:53:33.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:33.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0007aa0 msgr2=0x7f0ec0015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:33.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0007aa0 0x7f0ec0015c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:33.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.362+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0009790 legacy=0x7f0ec0009ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:33.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.563+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0007aa0 msgr2=0x7f0ec0015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:33.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.563+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0007aa0 0x7f0ec0015c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:33.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.563+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0009790 legacy=0x7f0ec0009ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:33.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.964+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0007aa0 msgr2=0x7f0ec0015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:33.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.964+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0007aa0 0x7f0ec0015c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:33.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:33.964+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0009790 legacy=0x7f0ec0009ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:34.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:34.765+0000 7f0edf7fe640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0007aa0 msgr2=0x7f0ec0015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:34.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:34.765+0000 7f0edf7fe640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0007aa0 0x7f0ec0015c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:34.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:34.765+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0009790 legacy=0x7f0ec0009ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:36.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:36.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:36.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:36.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:36.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0007aa0 msgr2=0x7f0ec0015c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:36.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec0007aa0 0x7f0ec0015c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:36.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec0009790 legacy=0x7f0ec0009ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:36.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-08-09T20:53:36.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:36.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:53:36.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec0010000 addr v2:172.21.15.194:3300/0 2024-08-09T20:53:36.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec00067c0 addr v1:172.21.15.194:6789/0 2024-08-09T20:53:36.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:36.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0006c50 addr v2:172.21.15.155:3300/0 2024-08-09T20:53:36.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec00067c0 2024-08-09T20:53:36.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:36.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0006c50 2024-08-09T20:53:36.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:36.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec0010000 2024-08-09T20:53:36.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:36.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edffff640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:36.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edffff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:36.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:36.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:36.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.362+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:36.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.563+0000 7f0edffff640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:36.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.563+0000 7f0edffff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:36.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.563+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:36.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.563+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:36.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.563+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:36.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.964+0000 7f0edffff640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:36.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.964+0000 7f0edffff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:36.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.964+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:36.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.964+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:36.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:36.964+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:37.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:37.765+0000 7f0edffff640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:37.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:37.765+0000 7f0edffff640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:37.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:37.765+0000 7f0edeffd640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:37.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:37.765+0000 7f0edf7fe640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:37.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:37.765+0000 7f0edf7fe640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:39.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:39.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:39.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:39.364 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:39.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:39.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:39.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:39.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:39.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:39.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[0,2] 2024-08-09T20:53:39.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec0007aa0 addr v1:172.21.15.155:6789/0 2024-08-09T20:53:39.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 0x7f0ec000a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:39.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec00107c0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:53:39.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ab5d0 con 0x7f0ec0007aa0 2024-08-09T20:53:39.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:39.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec00107c0 2024-08-09T20:53:39.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:39.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 msgr2=0x7f0ec000a7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:39.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:39.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.362+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 0x7f0ec000a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:39.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.563+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 msgr2=0x7f0ec000a7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:39.564 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.563+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:39.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.563+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 0x7f0ec000a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:39.965 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.964+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:39.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.964+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 msgr2=0x7f0ec000a7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:39.966 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:39.964+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 0x7f0ec000a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:40.766 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:40.765+0000 7f0edeffd640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 msgr2=0x7f0ec000a7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:40.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:40.765+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:40.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:40.765+0000 7f0edeffd640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 0x7f0ec000a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:42.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:42.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:42.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:42.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:42.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 msgr2=0x7f0ec000a7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:42.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 0x7f0ec000a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:42.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:42.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-08-09T20:53:42.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ec00067c0 addr v1:172.21.15.194:6789/0 2024-08-09T20:53:42.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edffff640 1 -- reap_dead start 2024-08-09T20:53:42.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:42.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec0010000 addr v2:172.21.15.194:3300/0 2024-08-09T20:53:42.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 0x7f0ec000a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:42.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec00107c0 addr [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] 2024-08-09T20:53:42.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 1 -- --> v1:172.21.15.194:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ada00 con 0x7f0ec00067c0 2024-08-09T20:53:42.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:42.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 1 -- --> v2:172.21.15.194:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0010000 2024-08-09T20:53:42.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:42.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 1 -- --> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec000af30 con 0x7f0ec00107c0 2024-08-09T20:53:42.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:42.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 msgr2=0x7f0ec000a7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:42.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 0x7f0ec000a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:42.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:42.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:42.368 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.363+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:42.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.564+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 msgr2=0x7f0ec000a7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:42.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.564+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 0x7f0ec000a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:42.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.564+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:42.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.564+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:42.566 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.564+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:42.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.965+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 msgr2=0x7f0ec000a7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:42.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.965+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 0x7f0ec000a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:42.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.965+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:42.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.965+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:42.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:42.965+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:43.767 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:43.766+0000 7f0edffff640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 msgr2=0x7f0ec000a7a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3301/0 2024-08-09T20:53:43.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:43.766+0000 7f0edffff640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 0x7f0ec000a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:43.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:43.766+0000 7f0edf7fe640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.194:6789/0 2024-08-09T20:53:43.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:43.766+0000 7f0edeffd640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.194:3300/0 2024-08-09T20:53:43.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:43.766+0000 7f0edeffd640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:45.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 10 monclient(hunting): tick 2024-08-09T20:53:45.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 10 monclient(hunting): _check_auth_tickets 2024-08-09T20:53:45.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 1 monclient(hunting): continuing hunt 2024-08-09T20:53:45.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 10 monclient(hunting): _reopen_session rank -1 2024-08-09T20:53:45.365 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 1 -- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 msgr2=0x7f0ec000a7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:45.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 1 --2- >> [v2:172.21.15.155:3301/0,v1:172.21.15.155:6790/0] conn(0x7f0ec00107c0 0x7f0ec000a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:45.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 1 -- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 msgr2=0x7f0ec000f680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:45.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.194:3300/0 conn(0x7f0ec0010000 0x7f0ec000f680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-08-09T20:53:45.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 1 -- >> v1:172.21.15.194:6789/0 conn(0x7f0ec00067c0 legacy=0x7f0ec0015010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-08-09T20:53:45.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 10 monclient: _add_conns ranks=[0,3] 2024-08-09T20:53:45.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ec0007aa0 addr v1:172.21.15.155:6789/0 2024-08-09T20:53:45.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-08-09T20:53:45.366 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec0006c50 addr v2:172.21.15.155:3300/0 2024-08-09T20:53:45.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 1 -- --> v1:172.21.15.155:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee00ab5d0 con 0x7f0ec0007aa0 2024-08-09T20:53:45.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 10 monclient(hunting): start opening mon connection 2024-08-09T20:53:45.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 1 -- --> v2:172.21.15.155:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee00ad020 con 0x7f0ec0006c50 2024-08-09T20:53:45.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edd7fa640 10 monclient(hunting): _renew_subs 2024-08-09T20:53:45.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:45.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:45.367 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.363+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-08-09T20:53:45.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.564+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:45.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.564+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:45.565 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.564+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-08-09T20:53:45.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.965+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:45.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.965+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:45.967 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:45.965+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-08-09T20:53:46.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:46.766+0000 7f0edf7fe640 1 -- >> v1:172.21.15.155:6789/0 conn(0x7f0ec0007aa0 legacy=0x7f0ec0015720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.155:6789/0 2024-08-09T20:53:46.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:46.766+0000 7f0edeffd640 1 -- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 msgr2=0x7f0ec0015e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.155:3300/0 2024-08-09T20:53:46.768 INFO:teuthology.orchestra.run.smithi155.stderr:2024-08-09T20:53:46.766+0000 7f0edeffd640 1 --2- >> v2:172.21.15.155:3300/0 conn(0x7f0ec0006c50 0x7f0ec0015e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-08-09T20:53:48.280 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-08-09T20:53:48.281 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 1923, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2777, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2609, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2024-08-09T20:53:48.360 INFO:teuthology.misc:Shutting down mds daemons... 2024-08-09T20:53:48.360 INFO:teuthology.misc:Shutting down osd daemons... 2024-08-09T20:53:48.360 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-08-09T20:53:48.361 INFO:teuthology.orchestra.run:waiting for 300 2024-08-09T20:53:48.364 INFO:tasks.ceph.osd.0:Stopped 2024-08-09T20:53:48.365 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-08-09T20:53:48.365 INFO:teuthology.orchestra.run:waiting for 300 2024-08-09T20:53:48.365 INFO:tasks.ceph.osd.1:Stopped 2024-08-09T20:53:48.365 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-08-09T20:53:48.365 INFO:teuthology.orchestra.run:waiting for 300 2024-08-09T20:53:48.366 INFO:tasks.ceph.osd.2:Stopped 2024-08-09T20:53:48.366 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-08-09T20:53:48.366 INFO:teuthology.orchestra.run:waiting for 300 2024-08-09T20:53:48.367 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-08-09T20:53:48.367 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 1442, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 1923, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2777, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2609, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-08-09T20:53:48.398 INFO:tasks.ceph.osd.3:Stopped 2024-08-09T20:53:48.398 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-08-09T20:53:48.398 INFO:teuthology.orchestra.run:waiting for 300 2024-08-09T20:53:48.398 INFO:tasks.ceph.osd.4:Stopped 2024-08-09T20:53:48.398 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-08-09T20:53:48.398 INFO:teuthology.orchestra.run:waiting for 300 2024-08-09T20:53:48.399 INFO:tasks.ceph.osd.5:Stopped 2024-08-09T20:53:48.399 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-08-09T20:53:48.399 INFO:teuthology.orchestra.run:waiting for 300 2024-08-09T20:53:48.399 INFO:tasks.ceph.osd.6:Stopped 2024-08-09T20:53:48.400 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-08-09T20:53:48.400 INFO:teuthology.orchestra.run:waiting for 300 2024-08-09T20:53:48.400 INFO:tasks.ceph.osd.7:Stopped 2024-08-09T20:53:48.400 INFO:teuthology.misc:Shutting down mgr daemons... 2024-08-09T20:53:48.400 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-08-09T20:53:48.400 INFO:teuthology.orchestra.run:waiting for 300 2024-08-09T20:53:48.401 INFO:tasks.ceph.mgr.y:Stopped 2024-08-09T20:53:48.401 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-08-09T20:53:48.401 INFO:teuthology.orchestra.run:waiting for 300 2024-08-09T20:53:48.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-09T20:53:48.401 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 1442, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 1923, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2777, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2609, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi194 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x' 2024-08-09T20:53:48.413 INFO:tasks.ceph.mgr.x:Stopped 2024-08-09T20:53:48.413 INFO:teuthology.misc:Shutting down mon daemons... 2024-08-09T20:53:48.413 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-08-09T20:53:48.413 INFO:teuthology.orchestra.run:waiting for 300 2024-08-09T20:53:48.414 INFO:tasks.ceph.mon.a:Stopped 2024-08-09T20:53:48.414 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-08-09T20:53:48.414 INFO:teuthology.orchestra.run:waiting for 300 2024-08-09T20:53:48.414 INFO:tasks.ceph.mon.c:Stopped 2024-08-09T20:53:48.415 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-08-09T20:53:48.415 INFO:teuthology.orchestra.run:waiting for 300 2024-08-09T20:53:48.415 INFO:tasks.ceph.mon.b:Stopped 2024-08-09T20:53:48.416 INFO:tasks.ceph:Checking cluster log for badness... 2024-08-09T20:53:48.416 DEBUG:teuthology.orchestra.run.smithi155:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-08-09T20:53:48.532 INFO:teuthology.orchestra.run.smithi155.stdout:2024-08-09T14:56:04.777390+0000 mon.a (mon.0) 698 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2024-08-09T20:53:48.533 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-08-09T20:53:48.533 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi155.front.sepia.ceph.com 2024-08-09T20:53:48.534 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-08-09T20:53:48.616 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi155.front.sepia.ceph.com 2024-08-09T20:53:48.616 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-08-09T20:53:48.689 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi155.front.sepia.ceph.com 2024-08-09T20:53:48.689 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-08-09T20:53:48.747 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi155.front.sepia.ceph.com 2024-08-09T20:53:48.747 DEBUG:teuthology.orchestra.run.smithi155:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-08-09T20:53:48.804 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi194.front.sepia.ceph.com 2024-08-09T20:53:48.805 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-08-09T20:53:48.960 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi194.front.sepia.ceph.com 2024-08-09T20:53:48.960 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-08-09T20:53:49.014 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi194.front.sepia.ceph.com 2024-08-09T20:53:49.015 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-08-09T20:53:49.064 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi194.front.sepia.ceph.com 2024-08-09T20:53:49.064 DEBUG:teuthology.orchestra.run.smithi194:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-08-09T20:53:49.117 INFO:tasks.ceph:Archiving mon data... 2024-08-09T20:53:49.118 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi/7845532/data/mon.a.tgz 2024-08-09T20:53:49.119 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-08-09T20:53:49.135 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.CJke4xRzAQ 2024-08-09T20:53:49.136 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.CJke4xRzAQ 2024-08-09T20:53:49.617 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.CJke4xRzAQ 2024-08-09T20:53:49.701 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.CJke4xRzAQ is 2MB 2024-08-09T20:53:49.825 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.CJke4xRzAQ 2024-08-09T20:53:49.840 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi/7845532/data/mon.c.tgz 2024-08-09T20:53:49.840 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-08-09T20:53:49.894 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.Yn3cZUAmeO 2024-08-09T20:53:49.895 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.Yn3cZUAmeO 2024-08-09T20:53:50.372 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.Yn3cZUAmeO 2024-08-09T20:53:50.455 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.Yn3cZUAmeO is 2MB 2024-08-09T20:53:50.550 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.Yn3cZUAmeO 2024-08-09T20:53:50.565 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi/7845532/data/mon.b.tgz 2024-08-09T20:53:50.565 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-08-09T20:53:50.584 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.DFSjjZJFv6 2024-08-09T20:53:50.585 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.DFSjjZJFv6 2024-08-09T20:53:51.210 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.DFSjjZJFv6 2024-08-09T20:53:51.293 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.DFSjjZJFv6 is 2MB 2024-08-09T20:53:51.412 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.DFSjjZJFv6 2024-08-09T20:53:51.428 INFO:tasks.ceph:Cleaning ceph cluster... 2024-08-09T20:53:51.428 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-09T20:53:51.431 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-08-09T20:53:51.793 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-08-09T20:53:52.029 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-08-09T20:53:52.029 INFO:tasks.ceph:Archiving crash dumps... 2024-08-09T20:53:52.031 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi/7845532/remote/smithi155/crash 2024-08-09T20:53:52.032 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-09T20:53:52.100 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi/7845532/remote/smithi194/crash 2024-08-09T20:53:52.101 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-08-09T20:53:52.124 INFO:tasks.ceph:Compressing logs... 2024-08-09T20:53:52.125 DEBUG:teuthology.orchestra.run.smithi155:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-09T20:53:52.127 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-09T20:53:52.152 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40131.log 2024-08-09T20:53:52.152 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40450.log 2024-08-09T20:53:52.154 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40706.log 2024-08-09T20:53:52.154 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40131.log: /var/log/ceph/ceph-client.admin.40450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42521.log 2024-08-09T20:53:52.155 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40706.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40131.log.gz 2024-08-09T20:53:52.155 INFO:teuthology.orchestra.run.smithi155.stderr: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.40450.log.gz -5 2024-08-09T20:53:52.155 INFO:teuthology.orchestra.run.smithi155.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41609.log 2024-08-09T20:53:52.155 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42521.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40706.log.gz 2024-08-09T20:53:52.155 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42636.log 2024-08-09T20:53:52.155 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42524.log 2024-08-09T20:53:52.155 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79058.log 2024-08-09T20:53:52.155 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100872.log 2024-08-09T20:53:52.158 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.79058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41768.log 2024-08-09T20:53:52.159 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.100872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100869.log 2024-08-09T20:53:52.159 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41042.log 2024-08-09T20:53:52.159 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.100869.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.79058.log.gz 2024-08-09T20:53:52.159 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-09T20:53:52.159 INFO:teuthology.orchestra.run.smithi155.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.42521.log.gz 2024-08-09T20:53:52.159 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42036.log 2024-08-09T20:53:52.160 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.41042.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41042.log.gz 2024-08-09T20:53:52.160 INFO:teuthology.orchestra.run.smithi155.stderr:gzip 90.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.100869.log.gz --verbose 2024-08-09T20:53:52.160 INFO:teuthology.orchestra.run.smithi155.stderr: -- /var/log/ceph/ceph-client.admin.35985.log 2024-08-09T20:53:52.161 INFO:teuthology.orchestra.run.smithi155.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.100872.log.gz 2024-08-09T20:53:52.163 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42036.log: 92.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-08-09T20:53:52.165 INFO:teuthology.orchestra.run.smithi155.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.42524.log.gz 2024-08-09T20:53:52.166 INFO:teuthology.orchestra.run.smithi155.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.41768.log.gz 2024-08-09T20:53:52.169 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41617.log 2024-08-09T20:53:52.169 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.35985.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-08-09T20:53:52.169 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40124.log 2024-08-09T20:53:52.169 INFO:teuthology.orchestra.run.smithi155.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35985.log.gz 2024-08-09T20:53:52.169 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42649.log 2024-08-09T20:53:52.169 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41701.log 2024-08-09T20:53:52.170 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39868.log 2024-08-09T20:53:52.170 INFO:teuthology.orchestra.run.smithi155.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40124.log.gz 2024-08-09T20:53:52.170 INFO:teuthology.orchestra.run.smithi155.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.42036.log.gz 2024-08-09T20:53:52.171 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41701.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42636.log.gz 2024-08-09T20:53:52.174 INFO:teuthology.orchestra.run.smithi155.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.41617.log.gz 2024-08-09T20:53:52.177 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108364.log 2024-08-09T20:53:52.177 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76357.log 2024-08-09T20:53:52.177 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.108364.log: gzip -5 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39868.log.gz 2024-08-09T20:53:52.177 INFO:teuthology.orchestra.run.smithi155.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42635.log 2024-08-09T20:53:52.178 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.76357.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-08-09T20:53:52.178 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41823.log 2024-08-09T20:53:52.178 INFO:teuthology.orchestra.run.smithi155.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.76357.log.gz 2024-08-09T20:53:52.178 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37937.log 2024-08-09T20:53:52.179 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.41823.log: 95.3% -- replaced with /var/log/ceph/ceph-client.admin.108364.log.gz 2024-08-09T20:53:52.179 INFO:teuthology.orchestra.run.smithi155.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42635.log.gz 2024-08-09T20:53:52.181 INFO:teuthology.orchestra.run.smithi155.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.42649.log.gz 2024-08-09T20:53:52.183 INFO:teuthology.orchestra.run.smithi155.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.41823.log.gz 2024-08-09T20:53:52.184 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41705.log 2024-08-09T20:53:52.184 INFO:teuthology.orchestra.run.smithi155.stderr: 95.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-08-09T20:53:52.185 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.37937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67165.log 2024-08-09T20:53:52.191 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41705.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37937.log.gz 2024-08-09T20:53:52.191 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79059.log 2024-08-09T20:53:52.191 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42268.log 2024-08-09T20:53:52.191 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.67165.log: /var/log/ceph/ceph-client.admin.79059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41841.log 2024-08-09T20:53:52.191 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40130.log 2024-08-09T20:53:52.192 INFO:teuthology.orchestra.run.smithi155.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42268.log.gz 2024-08-09T20:53:52.192 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40776.log 2024-08-09T20:53:52.192 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40130.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.67165.log.gz 2024-08-09T20:53:52.192 INFO:teuthology.orchestra.run.smithi155.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40130.log.gz 2024-08-09T20:53:52.192 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40666.log 2024-08-09T20:53:52.192 INFO:teuthology.orchestra.run.smithi155.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.79059.log.gz 2024-08-09T20:53:52.192 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42346.log 2024-08-09T20:53:52.193 INFO:teuthology.orchestra.run.smithi155.stderr: 87.8%/var/log/ceph/ceph-client.admin.40666.log: -- replaced with /var/log/ceph/ceph-client.admin.40776.log.gz 2024-08-09T20:53:52.193 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-08-09T20:53:52.193 INFO:teuthology.orchestra.run.smithi155.stderr:gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40666.log.gz 2024-08-09T20:53:52.193 INFO:teuthology.orchestra.run.smithi155.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42350.log 2024-08-09T20:53:52.193 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.42346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40133.log 2024-08-09T20:53:52.197 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42350.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.41841.log.gz 2024-08-09T20:53:52.197 INFO:teuthology.orchestra.run.smithi155.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.42350.log.gz 2024-08-09T20:53:52.198 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41515.log 2024-08-09T20:53:52.198 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41499.log 2024-08-09T20:53:52.198 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41515.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40133.log.gz 2024-08-09T20:53:52.198 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40904.log 2024-08-09T20:53:52.200 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41499.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.42346.log.gz 2024-08-09T20:53:52.201 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67164.log 2024-08-09T20:53:52.201 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40904.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40904.log.gz 2024-08-09T20:53:52.201 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40132.log 2024-08-09T20:53:52.209 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35344.log 2024-08-09T20:53:52.210 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-08-09T20:53:52.210 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35384.log 2024-08-09T20:53:52.210 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-08-09T20:53:52.210 INFO:teuthology.orchestra.run.smithi194.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35344.log.gz 2024-08-09T20:53:52.210 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35384.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-08-09T20:53:52.210 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-08-09T20:53:52.211 INFO:teuthology.orchestra.run.smithi194.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35384.log.gz 2024-08-09T20:53:52.211 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35544.log 2024-08-09T20:53:52.211 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-08-09T20:53:52.211 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35504.log 2024-08-09T20:53:52.211 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- 87.7% /var/log/ceph/ceph-client.admin.35293.log 2024-08-09T20:53:52.211 INFO:teuthology.orchestra.run.smithi194.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35544.log.gz 2024-08-09T20:53:52.211 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35504.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-08-09T20:53:52.212 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35293.log: gzip -5 --verbose 88.5% -- /var/log/ceph/valgrind/mon.b.log 2024-08-09T20:53:52.212 INFO:teuthology.orchestra.run.smithi194.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35504.log.gz 2024-08-09T20:53:52.212 INFO:teuthology.orchestra.run.smithi194.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35293.log.gz 2024-08-09T20:53:52.212 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-08-09T20:53:52.212 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/valgrind/mon.b.log: 73.5% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-08-09T20:53:52.212 INFO:teuthology.orchestra.run.smithi194.stderr: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-08-09T20:53:52.212 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-08-09T20:53:52.212 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 64.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-08-09T20:53:52.212 INFO:teuthology.orchestra.run.smithi194.stderr: 91.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-08-09T20:53:52.213 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T20:53:52.213 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-08-09T20:53:52.213 INFO:teuthology.orchestra.run.smithi194.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-08-09T20:53:52.213 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35424.log 2024-08-09T20:53:52.213 INFO:teuthology.orchestra.run.smithi194.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-08-09T20:53:52.213 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-08-09T20:53:52.213 INFO:teuthology.orchestra.run.smithi194.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-08-09T20:53:52.213 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35464.log 2024-08-09T20:53:52.214 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-08-09T20:53:52.214 INFO:teuthology.orchestra.run.smithi194.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.35424.log.gz 2024-08-09T20:53:52.214 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35584.log 2024-08-09T20:53:52.214 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-mon.b.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35464.log.gz 2024-08-09T20:53:52.214 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/ceph-client.admin.35584.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35584.log.gz 2024-08-09T20:53:52.214 INFO:teuthology.orchestra.run.smithi194.stderr: 94.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-08-09T20:53:52.214 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.67164.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.67164.log.gz 2024-08-09T20:53:52.215 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42675.log 2024-08-09T20:53:52.215 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40132.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40132.log.gz 2024-08-09T20:53:52.215 INFO:teuthology.orchestra.run.smithi155.stderr: 92.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40134.log 2024-08-09T20:53:52.215 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41499.log.gz 2024-08-09T20:53:52.215 INFO:teuthology.orchestra.run.smithi155.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.41515.log.gz 2024-08-09T20:53:52.215 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100867.log 2024-08-09T20:53:52.215 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42105.log 2024-08-09T20:53:52.215 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.100867.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40134.log.gz 2024-08-09T20:53:52.216 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-08-09T20:53:52.216 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42105.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.100867.log.gz 2024-08-09T20:53:52.217 INFO:teuthology.orchestra.run.smithi155.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.42105.log.gz 2024-08-09T20:53:52.217 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-08-09T20:53:52.217 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42164.log 2024-08-09T20:53:52.226 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39738.log 2024-08-09T20:53:52.229 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42164.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.42164.log.gz 2024-08-09T20:53:52.240 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-08-09T20:53:52.241 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39738.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39738.log.gz 2024-08-09T20:53:52.252 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40884.log 2024-08-09T20:53:52.266 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76356.log 2024-08-09T20:53:52.267 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40884.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40884.log.gz 2024-08-09T20:53:52.272 INFO:teuthology.orchestra.run.smithi155.stderr: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.42675.log.gz 2024-08-09T20:53:52.273 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-08-09T20:53:52.276 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.76356.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.76356.log.gz 2024-08-09T20:53:52.281 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41972.log 2024-08-09T20:53:52.295 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-08-09T20:53:52.305 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41972.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-08-09T20:53:52.305 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/valgrind/osd.0.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-08-09T20:53:52.314 INFO:teuthology.orchestra.run.smithi155.stderr: 91.7%gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-08-09T20:53:52.318 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/valgrind/mon.a.log: -- replaced with /var/log/ceph/ceph-client.admin.41972.log.gz 2024-08-09T20:53:52.323 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-08-09T20:53:52.327 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/valgrind/osd.3.log: 74.0% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-08-09T20:53:52.331 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-08-09T20:53:52.336 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/valgrind/osd.1.log: 80.9% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-08-09T20:53:52.343 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-08-09T20:53:52.345 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/valgrind/mon.c.log: 73.5% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-08-09T20:53:52.345 INFO:teuthology.orchestra.run.smithi155.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-08-09T20:53:52.353 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42353.log 2024-08-09T20:53:52.354 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/valgrind/osd.2.log: 64.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-08-09T20:53:52.362 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40400.log 2024-08-09T20:53:52.375 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87933.log 2024-08-09T20:53:52.376 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40400.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40400.log.gz 2024-08-09T20:53:52.382 INFO:teuthology.orchestra.run.smithi155.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.42353.log.gz 2024-08-09T20:53:52.383 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41478.log 2024-08-09T20:53:52.393 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.87933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41460.log 2024-08-09T20:53:52.397 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41478.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.87933.log.gz 2024-08-09T20:53:52.402 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-08-09T20:53:52.415 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36088.log 2024-08-09T20:53:52.424 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40881.log 2024-08-09T20:53:52.429 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.36088.log: 90.7% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-08-09T20:53:52.437 INFO:teuthology.orchestra.run.smithi155.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36088.log.gz 2024-08-09T20:53:52.441 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42517.log 2024-08-09T20:53:52.442 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40881.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40881.log.gz 2024-08-09T20:53:52.453 INFO:teuthology.orchestra.run.smithi155.stderr: 95.3% -- replaced with /var/log/ceph/ceph-client.admin.41701.log.gz 2024-08-09T20:53:52.454 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40775.log 2024-08-09T20:53:52.463 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42653.log 2024-08-09T20:53:52.465 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40775.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40775.log.gz 2024-08-09T20:53:52.479 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76361.log 2024-08-09T20:53:52.482 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42653.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.42653.log.gz 2024-08-09T20:53:52.483 INFO:teuthology.orchestra.run.smithi155.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.41460.log.gz 2024-08-09T20:53:52.483 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41538.log 2024-08-09T20:53:52.485 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.76361.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.76361.log.gz 2024-08-09T20:53:52.486 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40421.log 2024-08-09T20:53:52.491 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41538.log: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.41538.log.gz 2024-08-09T20:53:52.492 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41086.log 2024-08-09T20:53:52.492 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40421.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40421.log.gz 2024-08-09T20:53:52.493 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39817.log 2024-08-09T20:53:52.494 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41086.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41086.log.gz 2024-08-09T20:53:52.494 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40417.log 2024-08-09T20:53:52.496 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39817.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.39817.log.gz 2024-08-09T20:53:52.496 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67166.log 2024-08-09T20:53:52.497 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40417.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40417.log.gz 2024-08-09T20:53:52.498 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42340.log 2024-08-09T20:53:52.501 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.67166.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.67166.log.gz 2024-08-09T20:53:52.502 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39659.log 2024-08-09T20:53:52.508 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42340.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.42340.log.gz 2024-08-09T20:53:52.509 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40129.log 2024-08-09T20:53:52.511 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39659.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39659.log.gz 2024-08-09T20:53:52.511 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41945.log 2024-08-09T20:53:52.512 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40129.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40129.log.gz 2024-08-09T20:53:52.512 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42575.log 2024-08-09T20:53:52.520 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41945.log: 92.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88991.log 2024-08-09T20:53:52.525 INFO:teuthology.orchestra.run.smithi155.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.41945.log.gz 2024-08-09T20:53:52.525 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42517.log.gz 2024-08-09T20:53:52.537 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42349.log 2024-08-09T20:53:52.541 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.88991.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.88991.log.gz 2024-08-09T20:53:52.541 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87932.log 2024-08-09T20:53:52.548 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42349.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.42349.log.gz 2024-08-09T20:53:52.555 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40419.log 2024-08-09T20:53:52.572 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.87932.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.87932.log.gz 2024-08-09T20:53:52.574 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39794.log 2024-08-09T20:53:52.575 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40419.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40419.log.gz 2024-08-09T20:53:52.587 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41675.log 2024-08-09T20:53:52.588 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39794.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.39794.log.gz 2024-08-09T20:53:52.599 INFO:teuthology.orchestra.run.smithi155.stderr: 94.9% -- replaced with /var/log/ceph/ceph-client.admin.41705.log.gz 2024-08-09T20:53:52.599 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88992.log 2024-08-09T20:53:52.606 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41675.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.41675.log.gz 2024-08-09T20:53:52.615 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41849.log 2024-08-09T20:53:52.617 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.88992.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.88992.log.gz 2024-08-09T20:53:52.627 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40414.log 2024-08-09T20:53:52.643 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42030.log 2024-08-09T20:53:52.657 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40414.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40414.log.gz 2024-08-09T20:53:52.658 INFO:teuthology.orchestra.run.smithi155.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.41849.log.gz 2024-08-09T20:53:52.672 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42466.log 2024-08-09T20:53:52.692 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42328.log 2024-08-09T20:53:52.693 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42466.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.42466.log.gz 2024-08-09T20:53:52.701 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35682.log 2024-08-09T20:53:52.710 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88995.log 2024-08-09T20:53:52.713 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.35682.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.35682.log.gz 2024-08-09T20:53:52.723 INFO:teuthology.orchestra.run.smithi155.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.42328.log.gz 2024-08-09T20:53:52.724 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88994.log 2024-08-09T20:53:52.727 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.88995.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.88995.log.gz 2024-08-09T20:53:52.735 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39910.log 2024-08-09T20:53:52.743 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.88994.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.88994.log.gz 2024-08-09T20:53:52.747 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40472.log 2024-08-09T20:53:52.748 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39910.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39910.log.gz 2024-08-09T20:53:52.761 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42125.log 2024-08-09T20:53:52.763 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40472.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40472.log.gz 2024-08-09T20:53:52.775 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42537.log 2024-08-09T20:53:52.791 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42125.log: 93.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41002.log 2024-08-09T20:53:52.796 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42537.log: 95.2% -- replaced with /var/log/ceph/ceph-client.admin.41478.log.gz 2024-08-09T20:53:52.798 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42125.log.gz 2024-08-09T20:53:52.805 INFO:teuthology.orchestra.run.smithi155.stderr: 96.0% -- replaced with /var/log/ceph/ceph-client.admin.42575.log.gz 2024-08-09T20:53:52.807 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39952.log 2024-08-09T20:53:52.808 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41002.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41002.log.gz 2024-08-09T20:53:52.815 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40726.log 2024-08-09T20:53:52.825 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40449.log 2024-08-09T20:53:52.829 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40726.log: 87.2% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39952.log.gz 2024-08-09T20:53:52.829 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40726.log.gz 2024-08-09T20:53:52.833 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42341.log 2024-08-09T20:53:52.846 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41170.log 2024-08-09T20:53:52.847 INFO:teuthology.orchestra.run.smithi155.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40449.log.gz 2024-08-09T20:53:52.851 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42341.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.42341.log.gz 2024-08-09T20:53:52.861 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41975.log 2024-08-09T20:53:52.870 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41170.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36947.log 2024-08-09T20:53:52.874 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41975.log: -- replaced with /var/log/ceph/ceph-client.admin.41170.log.gz 2024-08-09T20:53:52.882 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42348.log 2024-08-09T20:53:52.884 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.36947.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.36947.log.gz 2024-08-09T20:53:52.894 INFO:teuthology.orchestra.run.smithi155.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.41975.log.gz 2024-08-09T20:53:52.901 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39254.log 2024-08-09T20:53:52.904 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42348.log: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.42348.log.gz 2024-08-09T20:53:52.918 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29816.log 2024-08-09T20:53:52.919 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39254.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39254.log.gz 2024-08-09T20:53:52.927 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42137.log 2024-08-09T20:53:52.936 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph.tmp-client.admin.29816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79057.log 2024-08-09T20:53:52.939 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.42137.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29816.log.gz 2024-08-09T20:53:52.946 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39603.log 2024-08-09T20:53:52.956 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.79057.log: 90.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41574.log 2024-08-09T20:53:52.963 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.39603.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79057.log.gz 2024-08-09T20:53:52.969 INFO:teuthology.orchestra.run.smithi155.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39603.log.gz 2024-08-09T20:53:52.974 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41128.log 2024-08-09T20:53:52.988 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41574.log: 95.0% -- replaced with /var/log/ceph/ceph-client.admin.42030.log.gz 2024-08-09T20:53:52.989 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36198.log 2024-08-09T20:53:52.989 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41128.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41128.log.gz 2024-08-09T20:53:52.991 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88993.log 2024-08-09T20:53:52.992 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.36198.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36198.log.gz 2024-08-09T20:53:52.992 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41691.log 2024-08-09T20:53:52.995 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.88993.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.88993.log.gz 2024-08-09T20:53:52.996 INFO:teuthology.orchestra.run.smithi155.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40125.log 2024-08-09T20:53:53.002 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.41691.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.41691.log.gz 2024-08-09T20:53:53.003 INFO:teuthology.orchestra.run.smithi155.stderr:/var/log/ceph/ceph-client.admin.40125.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40125.log.gz 2024-08-09T20:53:53.039 INFO:teuthology.orchestra.run.smithi155.stderr: 95.4% -- replaced with /var/log/ceph/ceph-client.admin.42537.log.gz 2024-08-09T20:53:53.054 INFO:teuthology.orchestra.run.smithi155.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.42137.log.gz 2024-08-09T20:53:53.305 INFO:teuthology.orchestra.run.smithi155.stderr: 95.1% -- replaced with /var/log/ceph/ceph-client.admin.41574.log.gz 2024-08-09T20:53:53.529 INFO:teuthology.orchestra.run.smithi155.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.41609.log.gz 2024-08-09T20:53:53.694 INFO:teuthology.orchestra.run.smithi194.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-08-09T20:53:53.708 INFO:teuthology.orchestra.run.smithi155.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-08-09T20:53:54.612 INFO:teuthology.orchestra.run.smithi155.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-08-09T20:53:55.420 INFO:teuthology.orchestra.run.smithi155.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-08-09T20:53:57.362 INFO:teuthology.orchestra.run.smithi194.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-08-09T20:53:57.576 INFO:teuthology.orchestra.run.smithi194.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-08-09T20:53:57.823 INFO:teuthology.orchestra.run.smithi155.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-08-09T20:53:57.850 INFO:teuthology.orchestra.run.smithi194.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-08-09T20:53:58.216 INFO:teuthology.orchestra.run.smithi194.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-08-09T20:53:58.218 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-08-09T20:53:58.218 INFO:teuthology.orchestra.run.smithi194.stderr:real 0m6.038s 2024-08-09T20:53:58.218 INFO:teuthology.orchestra.run.smithi194.stderr:user 0m22.931s 2024-08-09T20:53:58.219 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m0.811s 2024-08-09T20:53:58.249 INFO:teuthology.orchestra.run.smithi155.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-08-09T20:53:58.287 INFO:teuthology.orchestra.run.smithi155.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-08-09T20:53:58.289 INFO:teuthology.orchestra.run.smithi155.stderr: 2024-08-09T20:53:58.289 INFO:teuthology.orchestra.run.smithi155.stderr:real 0m6.148s 2024-08-09T20:53:58.289 INFO:teuthology.orchestra.run.smithi155.stderr:user 0m26.518s 2024-08-09T20:53:58.289 INFO:teuthology.orchestra.run.smithi155.stderr:sys 0m1.063s 2024-08-09T20:53:58.290 INFO:tasks.ceph:Archiving logs... 2024-08-09T20:53:58.290 DEBUG:teuthology.misc:Transferring archived files from smithi155:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi/7845532/remote/smithi155/log 2024-08-09T20:53:58.291 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-09T20:54:01.838 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi/7845532/remote/smithi194/log 2024-08-09T20:54:01.839 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-09T20:54:05.026 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 1923, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2777, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2609, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 1908, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-08-09T20:54:05.042 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-09T20:54:05.051 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 1923, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2777, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2609, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/task/install/__init__.py", line 641, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 1908, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-08-09T20:54:05.057 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-09T20:54:05.058 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-09T20:54:05.065 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-09T20:54:05.135 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-09T20:54:05.144 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-09T20:54:05.144 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-09T20:54:05.147 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-09T20:54:05.162 INFO:teuthology.orchestra.run.smithi155.stderr:bash: line 1: ntpq: command not found 2024-08-09T20:54:05.164 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-08-09T20:54:05.168 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-09T20:54:05.168 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2024-08-09T20:54:05.168 INFO:teuthology.orchestra.run.smithi155.stdout:^* hv01.front.sepia.ceph.com 3 8 377 120 -212us[ -209us] +/- 47ms 2024-08-09T20:54:05.169 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 3 +226us[ +226us] +/- 60ms 2024-08-09T20:54:05.169 INFO:teuthology.orchestra.run.smithi155.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 58 +105us[ +105us] +/- 87ms 2024-08-09T20:54:05.169 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-09T20:54:05.171 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-09T20:54:05.171 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-08-09T20:54:05.172 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv01.front.sepia.ceph.com 3 7 377 1 -243us[ -261us] +/- 49ms 2024-08-09T20:54:05.172 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv02.front.sepia.ceph.com 3 9 377 448 +196us[ +153us] +/- 62ms 2024-08-09T20:54:05.172 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 64 +123us[ +105us] +/- 87ms 2024-08-09T20:54:05.172 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-08-09T20:54:05.173 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-09T20:54:05.187 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-09T20:54:05.188 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-09T20:54:05.202 DEBUG:teuthology.orchestra.run.smithi155:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-09T20:54:05.277 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-09T20:54:05.345 DEBUG:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-09T20:54:05.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-09T20:54:05.375 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-09T20:54:05.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-09T20:54:05.405 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-09T20:54:05.415 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-09T20:54:05.424 INFO:teuthology.task.internal:Duration was 22714.933948 seconds 2024-08-09T20:54:05.425 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-09T20:54:05.434 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-09T20:54:05.434 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-09T20:54:05.437 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-09T20:54:05.476 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-09T20:54:05.486 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-09T20:54:05.792 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-09T20:54:05.792 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2024-08-09T20:54:05.793 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-09T20:54:05.854 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-08-09T20:54:05.854 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-09T20:54:05.879 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-09T20:54:05.879 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-09T20:54:05.896 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-09T20:54:05.946 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-09T20:54:05.946 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-09T20:54:05.964 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-09T20:54:06.201 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-09T20:54:06.579 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-09T20:54:06.579 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-09T20:54:06.607 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-09T20:54:06.635 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-09T20:54:06.646 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-09T20:54:06.654 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-09T20:54:06.674 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2024-08-09T20:54:06.701 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-08-09T20:54:06.751 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-09T20:54:06.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-09T20:54:06.769 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-09T20:54:06.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-09T20:54:06.811 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-09T20:54:06.824 INFO:teuthology.task.internal:Transferring archived files... 2024-08-09T20:54:06.825 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi/7845532/remote/smithi155 2024-08-09T20:54:06.825 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-09T20:54:06.956 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-08-09_13:31:16-rados-wip-yuri7-testing-2024-08-08-0804-squid-distro-default-smithi/7845532/remote/smithi194 2024-08-09T20:54:06.957 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-09T20:54:07.016 INFO:teuthology.task.internal:Removing archive directory... 2024-08-09T20:54:07.016 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-09T20:54:07.020 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-09T20:54:07.056 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-09T20:54:07.083 INFO:teuthology.task.internal:Not uploading archives. 2024-08-09T20:54:07.084 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-09T20:54:07.143 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-09T20:54:07.143 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-09T20:54:07.149 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-09T20:54:07.163 INFO:teuthology.orchestra.run.smithi155.stdout: 265306 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 9 20:54 /home/ubuntu/cephtest 2024-08-09T20:54:07.164 INFO:teuthology.orchestra.run.smithi155.stdout: 917831 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 9 14:50 /home/ubuntu/cephtest/mnt.0 2024-08-09T20:54:07.164 INFO:teuthology.orchestra.run.smithi155.stdout: 917832 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 9 14:51 /home/ubuntu/cephtest/mnt.0/client.0 2024-08-09T20:54:07.164 INFO:teuthology.orchestra.run.smithi155.stdout: 932325 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 9 14:51 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-09T20:54:07.164 INFO:teuthology.orchestra.run.smithi155.stdout: 932331 344 -rw-r--r-- 1 ubuntu ubuntu 348941 Aug 9 14:56 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_asio.log 2024-08-09T20:54:07.164 INFO:teuthology.orchestra.run.smithi155.stdout: 932336 4 -rw-r--r-- 1 ubuntu ubuntu 1243 Aug 9 14:52 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_pool.log 2024-08-09T20:54:07.164 INFO:teuthology.orchestra.run.smithi155.stdout: 932359 4 -rw-r--r-- 1 ubuntu ubuntu 631 Aug 9 15:00 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_list.log 2024-08-09T20:54:07.164 INFO:teuthology.orchestra.run.smithi155.stdout: 932351 4 -rw-r--r-- 1 ubuntu ubuntu 423 Aug 9 14:53 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_cls_remote_reads.log 2024-08-09T20:54:07.165 INFO:teuthology.orchestra.run.smithi155.stdout: 932333 8 -rw-r--r-- 1 ubuntu ubuntu 4986 Aug 9 14:58 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_misc.log 2024-08-09T20:54:07.165 INFO:teuthology.orchestra.run.smithi155.stdout: 932328 4 -rw-r--r-- 1 ubuntu ubuntu 3851 Aug 9 15:01 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_aio_pp.log 2024-08-09T20:54:07.165 INFO:teuthology.orchestra.run.smithi155.stdout: 932354 4 -rw-r--r-- 1 ubuntu ubuntu 585 Aug 9 14:57 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_cmd.log 2024-08-09T20:54:07.165 INFO:teuthology.orchestra.run.smithi155.stdout: 932348 8 -rw-r--r-- 1 ubuntu ubuntu 6649 Aug 9 14:52 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_service_pp.log 2024-08-09T20:54:07.165 INFO:teuthology.orchestra.run.smithi155.stdout: 932358 4 -rw-r--r-- 1 ubuntu ubuntu 519 Aug 9 14:56 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_ec_io.log 2024-08-09T20:54:07.165 INFO:teuthology.orchestra.run.smithi155.stdout: 932339 4 -rw-r--r-- 1 ubuntu ubuntu 2077 Aug 9 14:59 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_snapshots.log 2024-08-09T20:54:07.165 INFO:teuthology.orchestra.run.smithi155.stdout: 932342 4 -rw-r--r-- 1 ubuntu ubuntu 1178 Aug 9 14:55 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_stat.log 2024-08-09T20:54:07.165 INFO:teuthology.orchestra.run.smithi155.stdout: 932364 4 -rw-r--r-- 1 ubuntu ubuntu 1582 Aug 9 14:59 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_read_operations.log 2024-08-09T20:54:07.166 INFO:teuthology.orchestra.run.smithi155.stdout: 932365 4 -rw-r--r-- 1 ubuntu ubuntu 1703 Aug 9 14:58 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_snapshots.log 2024-08-09T20:54:07.166 INFO:teuthology.orchestra.run.smithi155.stdout: 932362 4 -rw-r--r-- 1 ubuntu ubuntu 365 Aug 9 14:51 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_misc.log 2024-08-09T20:54:07.166 INFO:teuthology.orchestra.run.smithi155.stdout: 932363 4 -rw-r--r-- 1 ubuntu ubuntu 931 Aug 9 14:52 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_pool.log 2024-08-09T20:54:07.166 INFO:teuthology.orchestra.run.smithi155.stdout: 932329 4 -rw-r--r-- 1 ubuntu ubuntu 2524 Aug 9 14:57 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_io.log 2024-08-09T20:54:07.166 INFO:teuthology.orchestra.run.smithi155.stdout: 932356 4 -rw-r--r-- 1 ubuntu ubuntu 508 Aug 9 14:56 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_cls.log 2024-08-09T20:54:07.166 INFO:teuthology.orchestra.run.smithi155.stdout: 932361 4 -rw-r--r-- 1 ubuntu ubuntu 400 Aug 9 14:57 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_ec_list.log 2024-08-09T20:54:07.167 INFO:teuthology.orchestra.run.smithi155.stdout: 932357 4 -rw-r--r-- 1 ubuntu ubuntu 481 Aug 9 14:52 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_handler_error.log 2024-08-09T20:54:07.167 INFO:teuthology.orchestra.run.smithi155.stdout: 932367 4 -rw-r--r-- 1 ubuntu ubuntu 985 Aug 9 15:00 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_write_operations.log 2024-08-09T20:54:07.167 INFO:teuthology.orchestra.run.smithi155.stdout: 932346 4 -rw-r--r-- 1 ubuntu ubuntu 363 Aug 9 14:51 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_cmd_pp.log 2024-08-09T20:54:07.167 INFO:teuthology.orchestra.run.smithi155.stdout: 932360 4 -rw-r--r-- 1 ubuntu ubuntu 1532 Aug 9 14:58 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_io.log 2024-08-09T20:54:07.167 INFO:teuthology.orchestra.run.smithi155.stdout: 932335 4 -rw-r--r-- 1 ubuntu ubuntu 1881 Aug 9 14:57 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_lock.log 2024-08-09T20:54:07.167 INFO:teuthology.orchestra.run.smithi155.stdout: 932350 4 -rw-r--r-- 1 ubuntu ubuntu 1206 Aug 9 14:59 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_c_write_operations.log 2024-08-09T20:54:07.167 INFO:teuthology.orchestra.run.smithi155.stdout: 932341 4 -rw-r--r-- 1 ubuntu ubuntu 3793 Aug 9 14:59 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_snapshots_pp.log 2024-08-09T20:54:07.168 INFO:teuthology.orchestra.run.smithi155.stdout: 932347 4 -rw-r--r-- 1 ubuntu ubuntu 2075 Aug 9 14:52 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_service.log 2024-08-09T20:54:07.168 INFO:teuthology.orchestra.run.smithi155.stdout: 932334 4 -rw-r--r-- 1 ubuntu ubuntu 2028 Aug 9 14:57 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_lock_pp.log 2024-08-09T20:54:07.168 INFO:teuthology.orchestra.run.smithi155.stdout: 932349 4 -rw-r--r-- 1 ubuntu ubuntu 1723 Aug 9 14:57 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_c_read_operations.log 2024-08-09T20:54:07.168 INFO:teuthology.orchestra.run.smithi155.stdout: 932332 8 -rw-r--r-- 1 ubuntu ubuntu 5508 Aug 9 14:57 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_list.log 2024-08-09T20:54:07.168 INFO:teuthology.orchestra.run.smithi155.stdout: 932330 4 -rw-r--r-- 1 ubuntu ubuntu 3924 Aug 9 14:57 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_io_pp.log 2024-08-09T20:54:07.168 INFO:teuthology.orchestra.run.smithi155.stdout: 932337 4 -rw-r--r-- 1 ubuntu ubuntu 2001 Aug 9 15:01 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_misc_pp.log 2024-08-09T20:54:07.168 INFO:teuthology.orchestra.run.smithi155.stdout: 932345 4 -rw-r--r-- 1 ubuntu ubuntu 2815 Aug 9 14:58 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_watch_notify.log 2024-08-09T20:54:07.169 INFO:teuthology.orchestra.run.smithi155.stdout: 932366 4 -rw-r--r-- 1 ubuntu ubuntu 808 Aug 9 14:53 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_neorados_watch_notify.log 2024-08-09T20:54:07.169 INFO:teuthology.orchestra.run.smithi155.stdout: 932343 8 -rw-r--r-- 1 ubuntu ubuntu 4383 Aug 9 14:57 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_watch_notify_pp.log 2024-08-09T20:54:07.169 INFO:teuthology.orchestra.run.smithi155.stdout: 932340 4 -rw-r--r-- 1 ubuntu ubuntu 1282 Aug 9 14:57 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_stat_pp.log 2024-08-09T20:54:07.170 INFO:teuthology.orchestra.run.smithi155.stdout: 932344 20 -rw-r--r-- 1 ubuntu ubuntu 16929 Aug 9 14:58 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_cmd.log 2024-08-09T20:54:07.170 INFO:teuthology.orchestra.run.smithi155.stdout: 932338 8 -rw-r--r-- 1 ubuntu ubuntu 5046 Aug 9 15:00 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_tier_pp.log 2024-08-09T20:54:07.170 INFO:teuthology.orchestra.run.smithi155.stdout: 932353 4 -rw-r--r-- 1 ubuntu ubuntu 1029 Aug 9 14:59 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_open_pools_parallel.log 2024-08-09T20:54:07.170 INFO:teuthology.orchestra.run.smithi155.stdout: 932352 8 -rw-r--r-- 1 ubuntu ubuntu 5540 Aug 9 14:59 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_list_parallel.log 2024-08-09T20:54:07.170 INFO:teuthology.orchestra.run.smithi155.stdout: 932327 4 -rw-r--r-- 1 ubuntu ubuntu 3632 Aug 9 15:01 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_api_aio.log 2024-08-09T20:54:07.170 INFO:teuthology.orchestra.run.smithi155.stdout: 932355 4 -rw-r--r-- 1 ubuntu ubuntu 1295 Aug 9 14:58 /home/ubuntu/cephtest/mnt.0/client.0/tmp/ceph_test_rados_delete_pools_parallel.log 2024-08-09T20:54:07.170 INFO:teuthology.orchestra.run.smithi155.stdout: 265542 720732 -rw------- 1 root root 738463744 Aug 9 14:56 /home/ubuntu/cephtest/vgcore.35989 2024-08-09T20:54:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-09T20:54:07.173 INFO:teuthology.orchestra.run.smithi194.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Aug 9 20:54 /home/ubuntu/cephtest 2024-08-09T20:54:07.173 INFO:teuthology.orchestra.run.smithi155.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-09T20:54:07.173 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 1923, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2777, in wait_for_clean num_active_clean = self.get_num_active_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2609, in get_num_active_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 2375, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 1630, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph_manager.py", line 1621, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 1908, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0a88632b45f25ae38493667c7dbd155f6e59b499/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_PossiblyLost calloc __trans_list_add During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_43b88056bcf6a0548bdd2d0502e56290cbae52df/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi155 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-09T20:54:07.194 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-09T20:54:07.209 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-09T20:54:07.233 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: valgrind error: Leak_PossiblyLost calloc __trans_list_add 2024-08-09T20:54:07.239 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 22714.93394804001 failure_reason: 'Command failed (workunit test rados/test.sh) on smithi155 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a88632b45f25ae38493667c7dbd155f6e59b499 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e4350912ae044b9d8587cd7fce2d5015 status: fail success: false 2024-08-09T20:54:07.239 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-09T20:54:07.356 INFO:teuthology.run:FAIL