Zed Series Release Notes

v1.14.0

New Features

  • The Open vSwitch container image now uses a more centralized location at ghcr.io/vexxhost/docker-openvswitch. This provides better maintainability and a dedicated repository for the Open vSwitch container images. The image now uses a specific version tag (v3.3.6-2) for better reproducibility and stability.

Bug Fixes

  • Fixed the node-exporter Prometheus monitoring configuration by setting the nodeExporterSelector to filter metrics by job="node-exporter" label. This ensures that node-exporter dashboards and alerts correctly reference the appropriate metrics.

  • Fix OctaviaAmphoraNotOperational monitoring rule to exclude DELETED Amphora status.

Other Notes

  • The libvirt exporter image switch to use ghcr.io/inovex/prometheus-libvirt-exporter, offering greater stability and performance on libvirt metrics collection.

v1.13.18

Bug Fixes

  • Fixed containers failing to validate TLS certificates on Red Hat-based systems. The issue occurred when mounting the OpenSSL trusted certificate bundle (/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt) which uses the “TRUSTED CERTIFICATE” format that’s incompatible with Go applications. The configuration now uses the standard PEM format bundle (/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem) on Red Hat systems, which resolves certificate validation errors.

  • Fix OctaviaAmphoraNotReady monitoring rule to recognize both READY and ALLOCATED as valid Amphora statuses. Previously, the monitoring rule fired for Amphora instances in ALLOCATED status, which is a normal operational state. The monitoring rule now uses the name OctaviaAmphoraNotOperational to better reflect its purpose of detecting non-operational Amphora instances.

v1.13.16

Upgrade Notes

  • Upgrade CAPI and CAPO version to 1.10.5 and 0.12.4 respectively.

Bug Fixes

  • Switched Valkey and Redis exporter images to Bitnami legacy repository due to Bitnami retiring their main registry. The upstream Valkey images don’t work out of the box, so this serves as a temporary workaround.

  • Upgrade the libvirt Helm chart from 0.1.27 to 1.1.0 to address critical issues with pod termination on systems using newer kernels. The updated chart includes proper mounting of the misc cgroup controller, which resolves failures where pods were unable to terminate correctly. This fix ensures stable pod lifecycle management in environments with modern kernel versions.

v1.13.15

New Features

  • Add confluent-kafka Python package to OpenStack images to enable the use of Kafka for notifications.

Upgrade Notes

  • Bump Kubernetes collection from 2.0.1 to 2.3.2 fix bugs and add new features.

  • Bump the Cluster API driver for Magnum from 0.30.0 to 0.31.2 to improve stability, fix bugs and add new features.

Bug Fixes

  • The designate-producer service runs a single replica instead of three to avoid issues with DNS zone serial updates. This is a workaround until the service has proper centralized locking.

v1.13.14

New Features

  • Atmosphere previously deactivated the Keystone auth token cache due to bug https://tracker.ceph.com/issues/64094. This issue is now resolved upstream, making it safe to reactivate the cache in the new version of Ceph which includes the fix (18.2.7).

  • The OpenStack database exporter has been updated and the collection of Octavia metrics happens through it only.

  • Added alerting for amphoras to cover cases for when an Amphora becomes in ERROR state or not ready for an unexpected duration.

Upgrade Notes

  • The max_allowed_packet setting increased from 4M (the default in MySQL 5.x) to 16M to support larger queries. Because MySQL 8.x uses a new default of 64M, the configuration no longer specifies this setting.

Security Issues

  • Upgrade nginx ingress controller from 1.1.1 to 1.12.1 to fix CVE-2025-1097 CVE-2025-1098, CVE-2025-1974, CVE-2025-24513, CVE-2025-24514.

Bug Fixes

  • The [cinder]/auth_type configuration value wasn’t set resulting in the entire Cinder section not render in the configuration file, it is now set to password which will fully render the Cinder section for OpenStack Nova.

  • The Cluster API driver for Magnum has been bumped to 0.28.0 to improve stability, fix bugs and add new features.

  • Added a custom build of Cluster API driver for OpenStack which includes fixes unblocking upgrades of Magnum clusters created using a specific network or subnet configuration.

  • Corrected Cinder authentication configuration handling in Nova. Nova now respects authentication overrides defined in OpenStack Helm endpoints, such as openstack_helm_endpoints_nova_region_name.

  • Manila now uses Nova micro-version 2.60 by default. This change enables support for attaching multiple volumes to an instance.

  • Manila now connects to the internal Nova and Glance endpoints instead of the public ones. This improves performance and reduces reliance on external network paths.

  • Addressed an issue where instances not booted from volume would fail to resize. This issue was caused by a missing trailing newline in the SSH key, which led to misinterpretation of the key material during the resize operation. Adding proper handling of SSH keys ensures that the resize process works as intended for all instances.

  • Fixed the OAuth2 Proxy configuration to enable API access using valid JWT tokens without requiring interactive login. Previously, OAuth2 Proxy enforced login for all requests by default. This change lets the Alertmanager API and other services behind OAuth2 Proxy support programmatic access via JWT tokens.

  • Improve alert generation for load balancers that have a non-ACTIVE provisioning state despite an ONLINE operational state. Previously, if a load balancer was in a transitional state such as PENDING_UPDATE (provisioning_state) while still marked as ONLINE (operational_state), the gauge metric openstack_loadbalancer_loadbalancer_status{provisioning_status!="ACTIVE"} did not trigger an alert. This update addresses the issue by ensuring that alerts are properly generated in these scenarios.

  • Increased the liveness probe timeouts for the Percona XtraDB Cluster. The configuration now sets timeoutSeconds to 60 and failureThreshold to 100. This change helps the cluster remain responsive and prevents unnecessary restarts during prolonged operations.

  • Changed the liveness check from the MySQL exporter sidecar to a readiness check. The sidecar should wait indefinitely for the main containers and shouldn’t terminate database pods. Especially during long SST operations. This change improves the cluster’s stability during extended operations.

  • Resolve the issue where the QEMU VNC and API TLS certificate fails to renew, preventing access to the virtual machine (VM) console via the dashboard and causing live migration failures.

Other Notes

  • Add documentation about database backup and restore procedures.

v1.13.13

Bug Fixes

  • The Cluster API driver for Magnum has been bumped to 0.27.0 to improve stability, fix bugs and add new features.

v1.13.12

New Features

  • Introduced a new Rust-based binary ovsinit which focuses on handling the migration of IP addresses from a physical interface to an OVS bridge during the Neutron or OVN initialization process.

Upgrade Notes

  • Upgrade Cluster API driver for Magnum to 0.26.0.

Bug Fixes

  • During a Neutron or OVN initialization process, the routes assigned to the physical interface are now removed and added to the OVS bridge to maintain the connectivity of the host.

  • The Cluster API driver for Magnum has been bumped to 0.26.2 to address bugs around cluster deletion.

v1.13.11

New Features

  • The ovn-northd service did not have liveness probes enabled which can result in the pod failing readiness checks but not being automatically restarted. The liveness probe is now enabled by default which will restart any stuck ovn-northd processes.

  • Neutron now supports using the built-in DHCP agent when using OVN (Open Virtual Network) for cases when DHCP relay is necessary.

Bug Fixes

  • The [privsep_osbrick]/helper_command configuration value was not configured in both of the Cinder and Nova services, which lead to the inability to run certain CLI commands since it instead tried to do a plain sudo instead. This has been fixed by adding the missing helper command configuration to both services.

  • The dmidecode package which is required by the os-brick library for certain operations was not installed on the images that needed it, which can cause NVMe-oF discovery issues. The package has been added to all images that require it.

  • The nova user within the nova-ssh image was missing the SHELL build argument which would cause live & cold migrations to fail, this has been resolved by adding the missing build argument.

  • This fix introduces a kernel option to adjust aio-max-nr, ensuring that the system can handle more asynchronous I/O events, preventing VM startup failures related to AIO limits.

  • The Cluster API driver for Magnum is now configured to use the internal endpoints by default in order to avoid going through the ingress and leverage client-side load balancing.

v1.13.10

New Features

  • Add specific helm-toolkit patch on 0.2.78. This will allow DB drop and init job compatible with SQLAlchemy 2.0

  • Add support for Neutron policy check when perform port update with add address pairs. This will add a POST method /address-pair. It will check if both ports (to be paired) are created within same project. With this check, we can give non-admin user to operate address pair binding without risk on expose resource to other projects.

  • Introduced the ability to specify a prefix for image names. This allows for easier integration with image proxies and caching mechanisms, eliminating the need to maintain separate inventory overrides for each image.

  • The ovn-controller image is now being pre-pulled on the nodes prior to the Helm chart being deployed. This will help reduce the time it takes to switch over to the new version of the ovn-controller image.

Bug Fixes

  • Fixed an issue where the neutron-ironic-agent service failed to start.

  • When use OVS with DPDK, by default both OVS and OVN run with root user, this may cause issue that QEMU can’t write vhost user socket file in openvswitch runtime directory (/run/openvswitch). This has been fixed by config Open vSwitch and OVN componments to run with non root user id 42424 which is same with QEMU and other OpenStack services inside the container.

  • The CI tooling for pinning images has been fixed to properly work after a regression caused by the introduction of the atmosphere_image_prefix variable.

  • The documentation for using the vTPM was pointing to the incorrect metadata properties for images. This has been corrected to point to the correct metadata properties.

Other Notes

  • The project has adopted the use of reno for release notes, ensuring that all changes include it from now on to ensure proper release notes.

  • The heavy CI jobs are now skipped when release notes are changed.

  • The image build process has been refactored to use docker-bake which allows us to use context/built images from one target to another, allowing for a much easier local building experience. There is no functional change in the images.