Summary

Recommendation ITU-T P.1203 provides the introductory document for a set of documents that describe model algorithms for monitoring the integral media session quality for TCP-type video streaming. The models comprise modules for short-term audio- and video-quality estimation. The per-one-second outputs of these short-term modules are integrated into estimates of audiovisual quality and, together with information about initial loading delay and media playout stalling events, further integrated into the final model output, the estimate of integral quality. The respective ITU-T work item has formerly been referred to as P.NATS (Parametric non-intrusive assessment of TCP-based multimedia streaming quality).

The structure of the set of recommendations reflects the different functionality of modules described in each document:

• ITU-T P.1203: Introductory document (this Recommendation)

• ITU-T P.1203.1: Video quality estimation module (short-term, providing per-one-second output information)

• ITU-T P.1203.2: Audio quality estimation module (short-term, providing per-one-second output information)

• ITU-T P.1203.3: Audiovisual integration and integration of final score, reflecting remembered quality for viewing sessions between 30 s and 5 min duration

The input used by the models consists of information obtained by prior stream inspection. Four different levels of inspection are included, resulting in models of different complexity both of the input information and the model algorithms, which are called "modes of operation" in the following:

• Mode 0: Information obtained from meta-information available during progressive download or adaptive streaming, for example from manifest files used in DASH, about codec and bitrate, and initial loading delay and stalling.

• Mode 1: All information from Mode 0, with additional video and audio frame information based on packet header inspection

• Mode 2: All information from Mode 1, and up to 2% (in Bytes) of the overall media stream information based on deep packet inspection and partial bitstream parsing

• Mode 3: All information from Mode 1, and complete media stream information based on bitstream parsing

The ITU-T P.1203-series of Recommendations addresses two application areas, which are respectively indicated in the module-related Recommendations [ITU-T P.1203.1], [ITU-T P.1203.2], [ITU-T P.1203.3]:

• Large-screen presentation as with fixed-network video streaming

• Mobile streaming on handheld devices such as smartphones

The ITU-T P.1203 module algorithms are no-reference, bitstream-based models.

History

1.  Scope

This Recommendation describes a set of objective parametric quality assessment modules that together can be used to form a complete model to predict the impact of audio and video media encodings and observed IP network impairments on quality experienced by the end-user in multimedia streaming applications. The addressed streaming techniques comprise progressive download as well as adaptive streaming, for both mobile and fixed network streaming applications.

Four modes are defined to cover a range of use-cases, from monitoring streams where the video payload is fully encrypted through to unencrypted streams, and where there are no limitations on processing power so that deep packet inspection is possible.

The model described is restricted to information provided to it by an appropriate bit stream analysis module or set of modules. The model described here is applicable to progressive download and adaptive streaming, where the quality experienced by the end user is affected by audio-coding and/or video degradations due to coding, spatial re-scaling or variations in video frame rates, as well as delivery degradations due to initial loading delay, stalling (which are both caused by rebuffering at the client), and media adaptations. Here, a "media adaptation" refers to events where the player switches video playback between a known set of media quality levels while adapting to network conditions. Each of the quality levels typically differs in a significant video and/or audio (and thus audiovisual) quality change. These quality changes are most readily observed by changes in bitrate, resolution, frame rate, and similar attributes.

The model predicts a mean opinion score (MOS) on a 5-point absolute category rating (ACR) scale (see [ITU-T P.910]) as a global multi-media MOS score (as defined in [ITU-T P.911], for instance). In addition, the well-defined modules provide several diagnostic outputs.

The primary applications for this model are monitoring of transmission quality for operations and maintenance purposes. The ITU-T P.1203 model for adaptive- and progressive-download-type media streaming may be deployed both in end-point locations and at mid-network monitoring points. Note, however, that the present Recommendation only describes the model that maps input parameters obtained from a probe located at a specific point in the network or in the client to the global multimedia MOS-scores and related quality diagnostic indicators, as described above.

The model associated with this Recommendation cannot provide a comprehensive evaluation of transmission quality as perceived by an individual end-user because its scores reflect the perceived impairments due to coded audiovisual media data being transmitted over an IP connection with certain performance, and does not include specific terminal devices or user-specific. The scores predicted by a general quality model necessarily reflect average perceptual impairments.

Effects such as those due to audio levels, signal noise and effects due to source generation such as video shake or certain colour properties (and other similar video or audio factors) and other impairments related to the payload are not reflected in the scores computed by this model. Therefore, it is possible to have high scores with this model, yet have a poor overall stream quality.

As a consequence, this Recommendation can be used for applications such as:

• In-service quality monitoring for specific IP-based audiovisual services, as specified in more detail below.

• Benchmarking of different service implementations. However, it cannot be used for direct benchmarking of different encoder implementations.

2.  References

3.  Definitions

5.  Conventions

This Recommendation uses the following conventions:

• Pa designates the audio quality estimation module [ITU-T P.1203.2].

• Pv designates the video quality estimation module (see [ITU-T P.1203.1]).

• Pq designates the quality integration module (see [ITU-T P.1203.3]).

6.  Areas of application

The application areas for ITU-T P.1203 are:

• Progressive download streaming and adaptive streaming (using reliable transport), which includes:

  • Over-the-top (OTT) services, as well as operator managed video services (over TCP).

  • Video over both mobile and fixed connections.

  • The protocols HTTP/TCP/IP, RTMP/TCP/IP, HLS/HTTP/TCP/IP, and DASH/HTTP/TCP/IP. Note that the model is agnostic to the specific network delivery method (HTTP or DASH or other), with one exception that it assumes reliable delivery (TCP/IP).

  • Video services typically using container formats such as Flash (FLV), MP4, WebM, 3GP, and MPEG2-TS. Note that the model is agnostic to the type of container format.

7.  Building blocks

The module layout of the ITU-T P.1203 model is depicted in Figure 1.

Figure 1 — Building blocks of the ITU-T P.1203 model

7.4.  Measurement window specification

The Pv [ITU-T P.1203.1] and Pa [ITU-T P.1203.2] modules provide one video or audio quality score per output sampling interval, respectively, and do not perform any kind of long‑term temporal integration of input features or output scores. This integration is handled in the integration module Pq specified in [ITU-T P.1203.3].

Pv and Pa modules must therefore apply a sliding measurement window for the input data acquisition and output score calculation. The measurement window is defined as:

audio or video information of one or more segments belonging to a specific media quality level, used as input to the Pv or Pa module at output time ts.

At any output time ts, the Pv ([ITU-T P.1203.1]) and Pa ([ITU-T P.1203.2]) modules can use information from the measurement window $\left[ts-T\text{/}2,ts+T\text{/}2\right]$, with T = 20 s, to generate the output sample according to the output sampling interval (see clause 7.5).

None of the following information must be used from outside the measurement window:

• bitstream data;

• previously calculated scores;

• extracted bitstream features, meta information, or any kind of indicators.

If the measurement window contains segments of multiple media quality levels, only contiguous adjacent segments of the same media quality level as the segment to which ts corresponds must be used as input to the Pa and Pv modules.

The timing of the measurement window input specification is visualized in Figure 2.

Figure 2 — Measurement window

7.4.1.  Implementation of measurement window

The measurement window must be implemented as described in clause 7.4.1.1 to clause 7.4.1.3.

7.4.1.1.  Frame extraction

From the input segments that form the sequence, each audio sample or video frame (depending on whether Pv or Pa is used) must be extracted (from now on simply called "frame"). Each frame must carry the information as described in the rows of Table 5, I.13 that are "per-frame", depending on the mode in which the module operates.

If frames cannot be extracted from physical bitstreams or video frame metadata (i.e., if the module is operating in mode 0), artificial frames must be generated by producing $S\cdot R$ frames for a segment of length S with a frame rate of R. Each frame must have a duration and decoding timestamp (DTS), with the DTS strictly monotonically increasing.

For example, for a segment of 5 s length with a frame rate of 25 Hz, 125 frames must be generated, with each frame having a duration of 0.04 s, and the frames having DTSs of $\left[0.04,0.08,0.12,\dots \right.$ ].

Note that artificial frames do not carry additional payload information.

7.4.1.2.  Determination of score calculation

An empty list must be initialized, which will hold frames. The last output time and the accumulated sequence duration must be set to 0. Then, for every frame extracted as described in clause 7.4.1.1, that frame is added to the list.

If the accumulated frame duration of the list is greater than 20 s, the first frame in the list is removed.

If the last output time is 0 and the accumulated sequence duration is smaller than 11, no score shall be output. Otherwise, if the accumulated sequence duration minus 10 is greater or equal to the last output time plus 1, the frame list and the corresponding output sample timestamp is forwarded to the Pa/Pv module to calculate the score, and the last output time is increased by 1.

The following pseudocode shows the procedure described above, which is called for every frame extracted:

if acc_frame_dur + frame.duration > 20:
    removed_frame = remove first item from frames
    acc_frame_dur -= removed_frame.duration

add frame to frames
acc_frame_dur += frame.duration
acc_sequence_dur += frame.duration

if last_score_output_at == 0 and acc_sequence_dur < 10 + 1:
    do nothing

if acc_sequence_dur - 10 >= last_score_output_at + 1:
    last_score_output_at += 1
    forward frames and output_sample_timestamp to module

When the last extracted frame has been added to the list and the above procedure has been run, the measurement window must be flushed according to the following procedure.

A final output sample timestamp is calculated by rounding down the total sequence duration (see clause 7.5). Then, repeatedly increasing the output sample timestamp by 1, frames that should not be considered are removed from the list, that is, if their DTS is lower than the current output sample timestamp minus 10. The remaining frames are then forwarded to the module for score calculation.

The following pseudocode explains the above procedure:

final_sample_timestamp = floor(acc_sequence_dur)
output_sample_timestamp = last_score_output_at + 1

while output_sample_timestamp <= final_sample_timestamp:
    while frames[0].dts < output_sample_timestamp - 10:
        remove first frame in frames

    forward frames and output_sample_timestamp to module

    output_sample_timestamp += 1

7.4.1.3.  Determination of contiguous adjacent segments

When the list of frames from the measurement window and the output sample timestamp are forwarded to the Pa or Pv module, the module has to determine which frames shall be considered for calculating the score at the output sample timestamp. It does so with the following procedure, where frames refers to the frames forwarded from the procedure in clause 7.4.1.2:

for index, frame in frames:
    if frame.dts < output_sample_timestamp:
        last_index = index
output_sample_relative = last_index

output_sample_frame = frames[output_sample_relative]
target_media_quality_level = output_sample_frame.media_quality_level
window = [output_sample_relative]

if frames[output_sample_relative-1].media_quality_level == target_media_quality_level:
    i = output_sample_relative - 1
    window = [i] + window
    if i-1 != -1:
        i -= 1
        while frames[i].media_quality_level == frames[i+1].media_quality_level:
            window = [i] + window
            if i-1==-1:
                break
            else:
                i-=1

if output_sample_relative + 1 != length(frames):
    if frames[output_sample_relative+1].media_quality_level ==
target_media_quality_level:
    i = output_sample_relative + 1
    window.append(i)
    if i+1 != len(frames):
        i += 1
        while frames[i].media_quality_level == frames[i-1].media_quality_level:
            window.append(i)
            if i+1 == len(frames):
                break
            else:
                i+=1

At the end of the procedure, "window" identifies the indices of frames (0-based) from the list of forwarded frames that must be used for calculating the quality score (see the respective procedures in [ITU-T P.1203.1] and [ITU-T P.1203.2].

8.  Overview of databases used for model development

For model development and validation, a total of 30 databases were created. Each database consists of a set of processed video sequences (PVSs). Within one database, each PVS was derived from a unique source video. The source videos of each database were of fixed duration in-between 1-5 minutes. The number of PVSs in each database was chosen such that the total video duration presented to subjects is around 60 minutes. In more detail, 60 PVSs were used for databases of 1‑minute duration, with fewer PVSs for the longer durations, and with a minimum of 14 PVSs for the source videos of 5-minute duration. In total, 1064 PVSs were used.

The source video sequences were processed by rescaling, encoding, and segmenting to form a set of quality representations for each video content segment. A processed video sequence was created by selecting one representation for each video content segment, with possibly introducing initial loading delay or stalling between the segments.

Video was encoded with [ITU-T H.264] using the libx264 codec with high10 profile and two-pass encoding, using different target bitrates. Scene cut detection was switched off and the maximum number of consecutive B-frames set to 3. The GOP duration was fixed for each video, but was variable in some of the databases.

Audio was encoded with AAC using the libfdk_aac codec.

For each database, an ACR-type subjective test was performed to collect ratings on the 5-point scale. Out of the 30 subjective tests, 19 were performed using a full-HD PC monitor for playback, and 11 were performed using a mobile phone with a 5-inch display.

Out of the 30 databases, 17 were initially shared for model development. The remaining 13 databases were used for model selection.

Overall performance p is determined by a weighted average of the per-database mean squared error (MSE). In more detail, the mean squared error MSE_k of database k is weighted by a weight w, summed over all databases and normalized,

where the weight w_k = 0.25 if the database k is part of the initially shared databases, and w = 0.75 otherwise. The total number of databases M is M = 30, and the normalisation constant N is given by $N=\sum _{k=1}^M{w}_k$.

9.  Description of the ITU-T P.1203 model algorithms

Detailed descriptions of the individual modules can be found in the respective Recommendations, and their annexes – [ITU-T P.1203.1] for the video quality estimation modules, [ITU-T P.1203.2] for the audio quality estimation module and [ITU-T P.1203.3] for the quality integration module.