The set of extensions known as Vulkan Video, developed by the Vulkan® Working Group at The Khronos® Group, provides developers with vendor-independent access to video decoding and encoding functionality in modern GPU hardware.

Today, with the release of version 1.4.321 of the Vulkan specification, Vulkan Video is once again being expanded for encoding operations with the introduction of the Encode Intra-refresh extension—the second advanced feature extension for encoding, following the earlier release of Encode Quantization Map.

Figure 1. Vulkan Video extensions

Encode Intra-refresh Extension

In today’s increasingly wireless world, intra-refresh is a valuable tool for enhancing video playback robustness in the presence of network errors. As a result, it is commonly used in broadcast streams, wireless video transmission, VoIP calls, and many other streaming applications.

Typically, each video frame is categorized as either a keyframe (which has no dependencies on other frames) or a predicted frame (which depends on other frames). If a keyframe or predicted frame cannot be correctly decoded—due to data corruption or dropped packets—persistent errors can occur due to inter-frame dependencies, as illustrated in Figure 2. This issue is especially severe when a keyframe fails to decode succesfully, as it is required to properly decode all subsequent predicted frames.

Figure 2. Dependency on corrupted previous frames causes persistent decoding errors

Intra-refresh repairs data corruption by gradually replacing predicted blocks with independently coded (intra-predicted) blocks over successive frames, limiting the spread of errors and eventually restoring clean reference data, as illustrated in Figure 3. Each frame in the input sequence is partitioned into intra-refresh regions, with all coding blocks in exactly one of these regions (highlighted in green) encoded using intra-prediction—thereby ‘refreshing’ that portion of the frame. The selected region is identified by its intra-refresh index. By incrementing this index across successive frames, the contents of all coding blocks are eventually refreshed.

The intra-refresh cycle is the complete set of frames required to refresh all regions of the input content. The final frame in the cycle is a clean frame, in which all coding blocks have been refreshed. Figure 3 illustrates an intra-refresh cycle lasting three frames. By repeating such cycles, the propagation of data errors can be significantly reduced.

Figure 3. Decode using Intra-Refresh mitigates against data corruption errors

Figure 3. illustrates intra-refresh regions being refreshed in a vertical pattern through the sequence of frames. Implementations may support additional intra-refresh modes, where intra-refresh regions are refreshed horizontally across a frame sequence.

More details on intra-refresh modes and other aspects of this extension can be found in the proposal document and specification.

Call for Action, Feedback & Support!

The Vulkan Working Group encourages developers to leverage the Vulkan Video extensions to unlock new levels of performance and functionality in their video applications. We welcome all feedback on the Encode Intra-refresh extension via the Vulkan issue tracker on GitHub and will continue to update issues with links to relevant Vulkan Video resources as they become available. We also look forward to hearing about additional features that are important to your use cases and should be considered for future Vulkan Video extensions.

An upcoming release of the Vulkan SDK will include updated Vulkan headers and Validation Layer support for the newly released video extensions. Meanwhile, you can find the Vulkan headers here.

Vulkan drivers supporting the Encode Intra-refresh extension, together with all previously released Vulkan Video extensions, are already available, including:

• NVIDIA: Windows and Linux beta drivers
• AMD: Windows beta driver
• Intel: Windows driver availability to follow

There is strong open-source community adoption of Vulkan Video, including decode and encode support in the GStreamer and FFmpeg multimedia frameworks, as well as the RADV/ANV open-source Vulkan drivers for AMD/Intel GPUs.

The Khronos open-source Vulkan Video sample is being updated to illustrate the use of the Encode Intra-refresh extension, and Igalia’s Vulkan Video status page provides an excellent compilation of Vulkan Video-related ecosystem resources.

We encourage your participation in shaping the Vulkan Video roadmap. See khronos.org/members to learn how to join Khronos and participate in the definition of any of our open standards.

Thank you for your interest in Vulkan Video. We hope you find it effective for your use cases and applications. We look forward to supporting your needs with additional codecs and features.