Attaching to a media element

There are distinct mechanisms for attaching a {{MediaSource}} to a media element depending on where the {{MediaSource}} object was constructed, in a {{Window}} versus in a {{DedicatedWorkerGlobalScope}}:

• Attaching a {{MediaSource}} that was constructed in a {{Window}} can be done by assigning a [=MediaSource object URL=] for that {{MediaSource}} to the media element {{HTMLMediaElement/src}} attribute or the src attribute of a <source> inside a media element. A [=MediaSource object URL=] is created by passing a MediaSource object to {{URL/createObjectURL()}}.

  Though implementations MAY allow [=MediaSource object URL=] creation in a {{DedicatedWorkerGlobalScope}} for a {{MediaSource}} constructed in that worker, attempting to use that [=MediaSource object URL=] to attach to a media element using either the {{HTMLMediaElement/src}} attribute or the src attribute of a <source> inside a media element MUST fail in the media element's [=resource fetch algorithm=], as extended below.

  Extending the object URL attachment mechanism to worker MediaSource object URLs would further propagate this idiom that is less preferred versus using srcObject, and would unnecessarily increase user agent interoperability risk and implementation complexity.

• Attaching a {{MediaSource}} that was constructed in a {{DedicatedWorkerGlobalScope}} can only be done by obtaining a handle from it using {{MediaSource/handle}}, transferring that {{MediaSourceHandle}} to the {{Window}} context and assigning it to the media element {{HTMLMediaElement/srcObject}} attribute. For the purposes of aligning this specification with {{HTMLMediaElement}} resource loading and fetching algorithms, the underlying {{DedicatedWorkerGlobalScope}} {{MediaSource}} is the MediaSource object mentioned there, and the {{MediaSourceHandle}} object is the media provider object.

If the [=resource fetch algorithm=] was invoked with a media provider object that is a {{MediaSource}} object, a {{MediaSourceHandle}} object or a URL record whose object is a {{MediaSource}} object, then let mode be local, skip the first step in the [=resource fetch algorithm=] (which may otherwise set mode to remote) and continue the execution of the [=resource fetch algorithm=].

The first step of the [=resource fetch algorithm=] is expected to eventually align with selecting local mode for URL records whose objects are media provider objects. The intent is that if the HTMLMediaElement's src attribute or selected child <source>'s src attribute is a blob: URL matching a [=MediaSource object URL=] when the respective src attribute was last changed, then that MediaSource object is used as the media provider object and current media resource in the local mode logic in the [=resource fetch algorithm=]. This also means that the remote mode logic that includes observance of any preload attribute is skipped when a MediaSource object is attached. Even with that eventual change to [[HTML]], the execution of the following steps at the beginning of the local mode logic is still required when the current media resource is a MediaSource object.

At the beginning of the "Otherwise (mode is local)" section of the [=resource fetch algorithm=], execute the additional steps, below.

Relative to the action which triggered the media element's resource selection algorithm, these steps are asynchronous. The [=resource fetch algorithm=] is run after the task that invoked the resource selection algorithm is allowed to continue and a stable state is reached. Implementations may delay the steps in the "Otherwise" clause, below, until the MediaSource object is ready for use.

1. If the [=resource fetch algorithm=] was invoked with a media provider object that is a {{MediaSource}} object, a {{MediaSourceHandle}} object or a URL record whose object is a {{MediaSource}} object, then:

   If the media provider object is a URL record whose object is a {{MediaSource}} that was constructed in a {{DedicatedWorkerGlobalScope}}, such as would occur if attempting to use a [=MediaSource object URL=] from a {{DedicatedWorkerGlobalScope}} {{MediaSource}}

   Run the steps of the [=resource fetch algorithm=]'s .

   This prevents using [=MediaSource object URLs=] for DedicatedWorker MediaSource attachments. Transferring {{MediaSource}}.{{MediaSource/handle}} from the DedicatedWorker to the Window context and assigning it to the media element's {{HTMLMediaElement/srcObject}} attribute is the only way to attach such a MediaSource.

   If the media provider object is a {{MediaSourceHandle}} whose {{MediaSourceHandle/[[Detached]]}} internal slot is true

   Run the steps of the [=resource fetch algorithm=]'s .

   If the media provider object is a {{MediaSourceHandle}} whose underlying {{MediaSource}}'s {{MediaSource/[[has ever been attached]]}} internal slot is true

   Run the steps of the [=resource fetch algorithm=]'s .

   This prevents loading an underlying {{MediaSource}} more than once using a {{MediaSourceHandle}}, even if the {{MediaSource}} was constructed on {{Window}} and had been loaded previously using a [=MediaSource object URL=]. This doesn't preclude subsequent use of a [=MediaSource object URL=] for a {{Window}} {{MediaSource}} from succeeding though.

   If {{MediaSource/readyState}} is NOT set to {{ReadyState/""closed""}}

   Run the steps of the [=resource fetch algorithm=]'s .

   Otherwise

   1. Set the {{MediaSource}}'s {{MediaSource/[[has ever been attached]]}} internal slot to true.
   2. Set the media element's to false.

   3. If the {{MediaSource}} was constructed in a {{DedicatedWorkerGlobalScope}}, then setup worker attachment communication and open the {{MediaSource}}:

      1. Set {{HTMLMediaElement/[[channel with worker]]}} to be a new {{MessageChannel}}.
      2. Set {{HTMLMediaElement/[[port to worker]]}} to the {{MessageChannel/port1}} value of {{HTMLMediaElement/[[channel with worker]]}}.
      3. Execute [=StructuredSerializeWithTransfer=] with the {{MessageChannel/port2}} of {{HTMLMediaElement/[[channel with worker]]}} as both the value and the sole member of the |transferList|, and let the result be |serialized port2:MessagePort|.
      4. [=Queue a task=] on the {{MediaSource}}'s {{DedicatedWorkerGlobalScope}} that will
         1. Execute [=StructuredDeserializeWithTransfer=] with |serialized port2| and {{DedicatedWorkerGlobalScope}}'s [=environment settings object/realm=], and set {{MediaSource/[[port to main]]}} to be the resulting deserialized clone of the transferred {{MessageChannel/port2}} value of {{HTMLMediaElement/[[channel with worker]]}}.
         2. Set the {{MediaSource/readyState}} attribute to {{ReadyState/""open""}}.
         3. [=Queue a task=] to [=fire an event=] named {{sourceopen}} at the MediaSource.

      Otherwise, the {{MediaSource}} was constructed in a {{Window}}:

      1. Set {{HTMLMediaElement/[[channel with worker]]}} null.
      2. Set {{HTMLMediaElement/[[port to worker]]}} null.
      3. Set {{MediaSource/[[port to main]]}} null.
      4. Set the {{MediaSource/readyState}} attribute to {{ReadyState/""open""}}.
      5. [=Queue a task=] to [=fire an event=] named {{sourceopen}} at the MediaSource.

   4. Continue the [=resource fetch algorithm=] by running the remaining steps, with these requirements:
      1. Text in the [=resource fetch algorithm=] or the that refers to "the download", "bytes received", or "whenever new data for the current media resource becomes available" refers to data passed in via {{SourceBuffer/appendBuffer()}}.
      2. References to HTTP in the [=resource fetch algorithm=] and the shall not apply because the HTMLMediaElement does not fetch media data via HTTP when a {{MediaSource}} is attached.

An attached MediaSource does not use the remote mode steps in the [=resource fetch algorithm=], so the media element will not fire "suspend" events. Though future versions of this specification will likely remove "progress" and "stalled" events from a media element with an attached MediaSource, user agents conforming to this version of the specification may still fire these two events as these [[HTML]] references changed after implementations of this specification stabilized.

Detaching from a media element

The following steps are run in any case where the media element is going to transition to {{HTMLMediaElement/NETWORK_EMPTY}} and [=queue a task=] to [=fire an event=] named [=HTMLMediaElement/emptied=] at the media element. These steps SHOULD be run right before the transition.

1. If the {{MediaSource}} was constructed in a {{DedicatedWorkerGlobalScope}}:

   1. Notify the {{MediaSource}} using an internal detach message posted to {{HTMLMediaElement/[[port to worker]]}}.
   2. Set {{HTMLMediaElement/[[port to worker]]}} null.
   3. Set {{HTMLMediaElement/[[channel with worker]]}} null.
   4. The implicit message handler for this detach notification runs the remainder of these steps in the {{DedicatedWorkerGlobalScope}} {{MediaSource}}.

   Otherwise, the {{MediaSource}} was constructed in a {{Window}}:

   Continue the remainder of these steps on the {{Window}} {{MediaSource}}.

2. Set {{MediaSource/[[port to main]]}} null.
3. Set the {{MediaSource/readyState}} attribute to {{ReadyState/""closed""}}.
4. If [=this=] is a {{ManagedMediaSource}}, then set {{ManagedMediaSource/streaming}} attribute to `false`.
5. Update {{MediaSource/duration}} to NaN.
6. Remove all the SourceBuffer objects from {{MediaSource/activeSourceBuffers}}.
7. [=Queue a task=] to [=fire an event=] named {{removesourcebuffer}} at {{MediaSource/activeSourceBuffers}}.
8. Remove all the SourceBuffer objects from {{MediaSource/sourceBuffers}}.
9. [=Queue a task=] to [=fire an event=] named {{removesourcebuffer}} at {{MediaSource/sourceBuffers}}.
10. [=Queue a task=] to [=fire an event=] named {{sourceclose}} at the MediaSource.

Going forward, this algorithm is intended to be externally called and run in any case where the attached MediaSource, if any, must be detached from the media element. It MAY be called on HTMLMediaElement [[HTML]] operations like load() and [=resource fetch algorithm=] failures in addition to, or in place of, when the media element transitions to {{HTMLMediaElement/NETWORK_EMPTY}}. Resource fetch algorithm failures are those which abort either the resource fetch algorithm or the resource selection algorithm, with the exception that the "Final step" [[HTML]] is not considered a failure that triggers detachment.

Seeking

Run the following steps as part of the "Wait until the user agent has established whether or not the media data for the new playback position is available, and, if it is, until it has decoded enough data to play back that position" step of the :

1. The media element looks for [=media segments=] containing the |new playback position:double| in each SourceBuffer object in {{MediaSource/activeSourceBuffers}}. Any position within a {{TimeRanges}} in the current value of the {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}} attribute has all necessary media segments buffered for that position.

   If |new playback position| is not in any {{TimeRanges}} of {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}}

   1. If the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute is greater than {{HTMLMediaElement/HAVE_METADATA}}, then set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to {{HTMLMediaElement/HAVE_METADATA}}.

      Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement.

   2. The media element waits until an {{SourceBuffer/appendBuffer()}} call causes the [=coded frame processing=] algorithm to set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to a value greater than {{HTMLMediaElement/HAVE_METADATA}}.

      The web application can use {{SourceBuffer/buffered}} and {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}} to determine what the media element needs to resume playback.

   Otherwise

   Continue

   If the {{MediaSource/readyState}} attribute is {{ReadyState/""ended""}} and the |new playback position| is within a {{TimeRanges}} currently in {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}}, then the seek operation must continue to completion here even if one or more currently selected or enabled track buffers' largest range end timestamp is less than |new playback position|. This condition should only occur due to logic in {{SourceBuffer/buffered}} when {{MediaSource/readyState}} is {{ReadyState/""ended""}}.

2. The media element resets all decoders and initializes each one with data from the appropriate [=initialization segment=].
3. The media element feeds [=coded frames=] from the [=active track buffers=] into the decoders starting with the closest [=random access point=] before the |new playback position|.
4. Resume the at the "Await a stable state" step.

SourceBuffer Monitoring

The following steps are periodically run during playback to make sure that all of the SourceBuffer objects in {{MediaSource/activeSourceBuffers}} have [=enough data to ensure uninterrupted playback=]. Changes to {{MediaSource/activeSourceBuffers}} also cause these steps to run because they affect the conditions that trigger state transitions.

Having enough data to ensure uninterrupted playback is an implementation specific condition where the user agent determines that it currently has enough data to play the presentation without stalling for a meaningful period of time. This condition is constantly evaluated to determine when to transition the media element into and out of the {{HTMLMediaElement/HAVE_ENOUGH_DATA}} ready state. These transitions indicate when the user agent believes it has enough data buffered or it needs more data respectively.

An implementation MAY choose to use bytes buffered, time buffered, the append rate, or any other metric it sees fit to determine when it has enough data. The metrics used MAY change during playback so web applications SHOULD only rely on the value of {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} to determine whether more data is needed or not.

When the media element needs more data, the user agent SHOULD transition it from {{HTMLMediaElement/HAVE_ENOUGH_DATA}} to {{HTMLMediaElement/HAVE_FUTURE_DATA}} early enough for a web application to be able to respond without causing an interruption in playback. For example, transitioning when the current playback position is 500ms before the end of the buffered data gives the application roughly 500ms to append more data before playback stalls.

If the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute equals {{HTMLMediaElement/HAVE_NOTHING}}:

1. Abort these steps.

If {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}} does not contain a {{TimeRanges}} for the current playback position:

1. Set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to {{HTMLMediaElement/HAVE_METADATA}}.

   Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement.

2. Abort these steps.

If {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}} contains a {{TimeRanges}} that includes the current playback position and [=enough data to ensure uninterrupted playback=]:

1. Set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to {{HTMLMediaElement/HAVE_ENOUGH_DATA}}.

   Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement.

2. Playback may resume at this point if it was previously suspended by a transition to {{HTMLMediaElement/HAVE_CURRENT_DATA}}.
3. Abort these steps.

If {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}} contains a {{TimeRanges}} that includes the current playback position and some time beyond the current playback position, then run the following steps:

1. Set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to {{HTMLMediaElement/HAVE_FUTURE_DATA}}.

   Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement.

2. Playback may resume at this point if it was previously suspended by a transition to {{HTMLMediaElement/HAVE_CURRENT_DATA}}.
3. Abort these steps.

If {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}} contains a {{TimeRanges}} that ends at the current playback position and does not have a range covering the time immediately after the current position:

1. Set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to {{HTMLMediaElement/HAVE_CURRENT_DATA}}.

   Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement.

2. Playback is suspended at this point since the media element doesn't have enough data to advance the .
3. Abort these steps.

Changes to selected/enabled track state

During playback {{MediaSource/activeSourceBuffers}} needs to be updated if the {{VideoTrack/selected}} video track, the {{AudioTrack/enabled}} audio track(s), or a text track {{TextTrack/mode}} changes. When one or more of these changes occur the following steps need to be followed. Also, when {{MediaSource}} was constructed in a {{DedicatedWorkerGlobalScope}}, then each change that occurs to a {{Window}} mirror of a track created previously by the implicit handler for the internal create track mirror message MUST also be made to the corresponding {{DedicatedWorkerGlobalScope}} track using an internal update track state message posted to {{HTMLMediaElement/[[port to worker]]}} whose implicit handler makes the change and runs the following steps. Likewise, each change that occurs to a {{DedicatedWorkerGlobalScope}} track MUST also be made to the corresponding {{Window}} mirror of the track using an internal update track state message posted to {{MediaSource/[[port to main]]}} whose implicit handler makes the change to the mirror.

If the selected video track changes, then run the following steps:

1. If the SourceBuffer associated with the previously selected video track is not associated with any other enabled tracks, run the following steps:
   1. Remove the SourceBuffer from {{MediaSource/activeSourceBuffers}}.
   2. [=Queue a task=] to [=fire an event=] named {{removesourcebuffer}} at {{MediaSource/activeSourceBuffers}}
2. If the SourceBuffer associated with the newly selected video track is not already in {{MediaSource/activeSourceBuffers}}, run the following steps:
   1. Add the SourceBuffer to {{MediaSource/activeSourceBuffers}}.
   2. [=Queue a task=] to [=fire an event=] named {{addsourcebuffer}} at {{MediaSource/activeSourceBuffers}}

If an audio track becomes disabled and the SourceBuffer associated with this track is not associated with any other enabled or selected track, then run the following steps:

1. Remove the SourceBuffer associated with the audio track from {{MediaSource/activeSourceBuffers}}
2. [=Queue a task=] to [=fire an event=] named {{removesourcebuffer}} at {{MediaSource/activeSourceBuffers}}

If an audio track becomes enabled and the SourceBuffer associated with this track is not already in {{MediaSource/activeSourceBuffers}}, then run the following steps:

1. Add the SourceBuffer associated with the audio track to {{MediaSource/activeSourceBuffers}}
2. [=Queue a task=] to [=fire an event=] named {{addsourcebuffer}} at {{MediaSource/activeSourceBuffers}}

If a text track {{TextTrack/mode}} becomes and the SourceBuffer associated with this track is not associated with any other enabled or selected track, then run the following steps:

1. Remove the SourceBuffer associated with the text track from {{MediaSource/activeSourceBuffers}}
2. [=Queue a task=] to [=fire an event=] named {{removesourcebuffer}} at {{MediaSource/activeSourceBuffers}}

If a text track {{TextTrack/mode}} becomes or and the SourceBuffer associated with this track is not already in {{MediaSource/activeSourceBuffers}}, then run the following steps:

1. Add the SourceBuffer associated with the text track to {{MediaSource/activeSourceBuffers}}
2. [=Queue a task=] to [=fire an event=] named {{addsourcebuffer}} at {{MediaSource/activeSourceBuffers}}

Duration change

Follow these steps when {{MediaSource/duration}} needs to change to a |new duration:unrestricted double|.

1. If the current value of {{MediaSource/duration}} is equal to |new duration|, then return.
2. If |new duration| is less than the highest [=presentation timestamp=] of any buffered [=coded frames=] for all SourceBuffer objects in {{MediaSource/sourceBuffers}}, then throw an {{InvalidStateError}} exception and abort these steps.

   Duration reductions that would truncate currently buffered media are disallowed. When truncation is necessary, use {{SourceBuffer/remove()}} to reduce the buffered range before updating {{MediaSource/duration}}.

3. Let |highest end time:unrestricted double| be the largest [=track buffer ranges=] end time across all the [=track buffers=] across all SourceBuffer objects in {{MediaSource/sourceBuffers}}.
4. If |new duration| is less than |highest end time|, then

   This condition can occur because the [=coded frame removal=] algorithm preserves coded frames that start before the start of the removal range.

   1. Update |new duration| to equal |highest end time|.
5. Update {{MediaSource/duration}} to |new duration|.
6. Use the [=mirror if necessary=] algorithm to run the following steps in {{Window}} to update the media element's duration:
   1. Update the media element's {{HTMLMediaElement/duration}} to |new duration|.
   2. Run the .

End of stream

This algorithm gets called when the application signals the end of stream via an {{MediaSource/endOfStream()}} call or an algorithm needs to signal a decode error. This algorithm takes an |error:EndOfStreamError| parameter that indicates whether an error will be signalled.

1. Change the {{MediaSource/readyState}} attribute value to {{ReadyState/""ended""}}.
2. [=Queue a task=] to [=fire an event=] named {{sourceended}} at the MediaSource.

3. If |error| is not set

   1. Run the [=duration change=] algorithm with |new duration:unrestricted double| set to the largest [=track buffer ranges=] end time across all the [=track buffers=] across all SourceBuffer objects in {{MediaSource/sourceBuffers}}.

      This allows the duration to properly reflect the end of the appended media segments. For example, if the duration was explicitly set to 10 seconds and only media segments for 0 to 5 seconds were appended before endOfStream() was called, then the duration will get updated to 5 seconds.

   2. Notify the media element that it now has all of the media data.

   If |error| is set to {{EndOfStreamError/""network""}}

   Use the [=mirror if necessary=] algorithm to run the following steps in {{Window}}:

   If the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute equals {{HTMLMediaElement/HAVE_NOTHING}}

   Run the steps of the [=resource fetch algorithm=]'s .

   If the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute is greater than {{HTMLMediaElement/HAVE_NOTHING}}

   Run the "If the connection is interrupted after some media data has been received, causing the user agent to give up trying to fetch the resource" steps of the [=resource fetch algorithm=]'s .

   If |error| is set to {{EndOfStreamError/""decode""}}

   Use the [=mirror if necessary=] algorithm to run the following steps in {{Window}}:

   If the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute equals {{HTMLMediaElement/HAVE_NOTHING}}

   Run the "If the media data can be fetched but is found by inspection to be in an unsupported format, or can otherwise not be rendered at all" steps of the [=resource fetch algorithm=]'s .

   If the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute is greater than {{HTMLMediaElement/HAVE_NOTHING}}

   Run the steps of the [=resource fetch algorithm=]'s .

Mirror if necessary

This algorithm is used to run steps on {{Window}} from a {{MediaSource}} attached from either the same {{Window}} or from a {{DedicatedWorkerGlobalScope}}, usually to update the state of the attached {{HTMLMediaElement}}. This algorithm takes a |steps| parameter that lists the steps to run on {{Window}}.

If the {{MediaSource}} was constructed in a {{DedicatedWorkerGlobalScope}}:

Post an internal mirror on window message to {{MediaSource/[[port to main]]}} whose implicit handler in {{Window}} will run |steps|. Return control to the caller without awaiting that handler's receipt of the message.

The purpose of the mirror message mechanism is to ensure that:

1. |steps| run asynchronously as their own task on {{Window}} rather than these |steps| somehow happening in the middle of some other {{Window}} task's execution, and
2. |steps| are run without blocking the synchronous execution and return of this algorithm on {{DedicatedWorkerGlobalScope}}.

Otherwise:

Run |steps|.

Segment Parser Loop

Each {{SourceBuffer}} object has an [[\append state]] internal slot that keeps track of the high-level segment parsing state. It is initially set to [=WAITING_FOR_SEGMENT=] and can transition to the following states as data is appended.

Each {{SourceBuffer}} object has an [[\input buffer]] internal slot that is a byte buffer that holds unparsed bytes across {{SourceBuffer/appendBuffer()}} calls. The buffer is empty when the {{SourceBuffer}} object is created.

Each {{SourceBuffer}} object has a [[\buffer full flag]] internal slot that keeps track of whether {{SourceBuffer/appendBuffer()}} is allowed to accept more bytes. It is set to false when the {{SourceBuffer}} object is created and gets updated as data is appended and removed.

Each {{SourceBuffer}} object has a [[\group start timestamp]] internal slot that keeps track of the starting timestamp for a new [=coded frame group=] in the {{AppendMode/""sequence""}} mode. It is unset when the SourceBuffer object is created and gets updated when the {{SourceBuffer/mode}} attribute equals {{AppendMode/""sequence""}} and the {{SourceBuffer/timestampOffset}} attribute is set, or the [=coded frame processing=] algorithm runs.

Each {{SourceBuffer}} object has a [[\group end timestamp]] internal slot that stores the highest [=coded frame end timestamp=] across all [=coded frames=] in the current [=coded frame group=]. It is set to 0 when the SourceBuffer object is created and gets updated by the [=coded frame processing=] algorithm.

The {{SourceBuffer/[[group end timestamp]]}} stores the highest [=coded frame end timestamp=] across all [=track buffers=] in a SourceBuffer. Therefore, care should be taken in setting the {{SourceBuffer/mode}} attribute when appending multiplexed segments in which the timestamps are not aligned across tracks.

Each {{SourceBuffer}} object has a [[\generate timestamps flag]] internal slot that is a boolean that keeps track of whether timestamps need to be generated for the [=coded frames=] passed to the [=coded frame processing=] algorithm. This flag is set by {{MediaSource/addSourceBuffer()}} when the {{SourceBuffer}} object is created and is updated by {{SourceBuffer/changeType()}}.

When the segment parser loop algorithm is invoked, run the following steps:

1. Loop Top: If the {{SourceBuffer/[[input buffer]]}} is empty, then jump to the need more data step below.
2. If the {{SourceBuffer/[[input buffer]]}} contains bytes that violate the [=SourceBuffer byte stream format specification=], then run the [=append error=] algorithm and abort this algorithm.
3. Remove any bytes that the [=byte stream format specifications=] say MUST be ignored from the start of the {{SourceBuffer/[[input buffer]]}}.

4. If the {{SourceBuffer/[[append state]]}} equals [=WAITING_FOR_SEGMENT=], then run the following steps:

   1. If the beginning of the {{SourceBuffer/[[input buffer]]}} indicates the start of an [=initialization segment=], set the {{SourceBuffer/[[append state]]}} to [=PARSING_INIT_SEGMENT=].
   2. If the beginning of the {{SourceBuffer/[[input buffer]]}} indicates the start of a [=media segment=], set {{SourceBuffer/[[append state]]}} to [=PARSING_MEDIA_SEGMENT=].
   3. Jump to the loop top step above.

5. If the {{SourceBuffer/[[append state]]}} equals [=PARSING_INIT_SEGMENT=], then run the following steps:

   1. If the {{SourceBuffer/[[input buffer]]}} does not contain a complete [=initialization segment=] yet, then jump to the need more data step below.
   2. Run the [=initialization segment received=] algorithm.
   3. Remove the [=initialization segment=] bytes from the beginning of the {{SourceBuffer/[[input buffer]]}}.
   4. Set {{SourceBuffer/[[append state]]}} to [=WAITING_FOR_SEGMENT=].
   5. Jump to the loop top step above.

6. If the {{SourceBuffer/[[append state]]}} equals [=PARSING_MEDIA_SEGMENT=], then run the following steps:

   1. If the {{SourceBuffer/[[first initialization segment received flag]]}} is false or the {{SourceBuffer/[[pending initialization segment for changeType flag]]}} is true, then run the [=append error=] algorithm and abort this algorithm.
   2. If the {{SourceBuffer/[[input buffer]]}} contains one or more complete [=coded frames=], then run the [=coded frame processing=] algorithm.

      The frequency at which the coded frame processing algorithm is run is implementation-specific. The coded frame processing algorithm MAY be called when the input buffer contains the complete media segment or it MAY be called multiple times as complete coded frames are added to the input buffer.

   3. If this {{SourceBuffer}} is full and cannot accept more media data, then set the {{SourceBuffer/[[buffer full flag]]}} to true.
   4. If the {{SourceBuffer/[[input buffer]]}} does not contain a complete [=media segment=], then jump to the need more data step below.
   5. Remove the [=media segment=] bytes from the beginning of the {{SourceBuffer/[[input buffer]]}}.
   6. Set {{SourceBuffer/[[append state]]}} to [=WAITING_FOR_SEGMENT=].
   7. Jump to the loop top step above.
7. Need more data: Return control to the calling algorithm.

Reset Parser State

When the parser state needs to be reset, run the following steps:

1. If the {{SourceBuffer/[[append state]]}} equals [=PARSING_MEDIA_SEGMENT=] and the {{SourceBuffer/[[input buffer]]}} contains some complete [=coded frames=], then run the [=coded frame processing=] algorithm until all of these complete [=coded frames=] have been processed.
2. Unset the [=last decode timestamp=] on all [=track buffers=].
3. Unset the [=last frame duration=] on all [=track buffers=].
4. Unset the [=highest end timestamp=] on all [=track buffers=].
5. Set the [=need random access point flag=] on all [=track buffers=] to true.
6. If the {{SourceBuffer/mode}} attribute equals {{AppendMode/""sequence""}}, then set the {{SourceBuffer/[[group start timestamp]]}} to the {{SourceBuffer/[[group end timestamp]]}}
7. Remove all bytes from the {{SourceBuffer/[[input buffer]]}}.
8. Set {{SourceBuffer/[[append state]]}} to [=WAITING_FOR_SEGMENT=].

Append Error

This algorithm is called when an error occurs during an append.

1. Run the [=reset parser state=] algorithm.
2. Set the {{SourceBuffer/updating}} attribute to false.
3. [=Queue a task=] to [=fire an event=] named {{error}} at this SourceBuffer object.
4. [=Queue a task=] to [=fire an event=] named {{updateend}} at this SourceBuffer object.
5. Run the [=end of stream=] algorithm with the |error:EndOfStreamError| parameter set to {{EndOfStreamError/""decode""}}.

Prepare Append

When an append operation begins, the following steps are run to validate and prepare the SourceBuffer.

1. If the SourceBuffer has been removed from the {{MediaSource/sourceBuffers}} attribute of the [=parent media source=] then throw an {{InvalidStateError}} exception and abort these steps.
2. If the {{SourceBuffer/updating}} attribute equals true, then throw an {{InvalidStateError}} exception and abort these steps.
3. Let |recent element error:boolean| be determined as follows:

   If the {{MediaSource}} was constructed in a {{Window}}

   Let |recent element error| be true if the {{HTMLMediaElement}}.{{HTMLMediaElement/error}} attribute is not null. If that attribute is null, then let |recent element error| be false.

   Otherwise

   Let |recent element error| be the value resulting from the steps for the {{Window}} case, but run on the {{Window}} {{HTMLMediaElement}} on any change to its {{HTMLMediaElement/error}} attribute and communicated by using {{HTMLMediaElement/[[port to worker]]}} implicit messages. If such a message has not yet been received, then let |recent element error| be false.

4. If |recent element error| is true, then throw an {{InvalidStateError}} exception and abort these steps.

5. If the {{MediaSource/readyState}} attribute of the [=parent media source=] is in the {{ReadyState/""ended""}} state then run the following steps:

   1. Set the {{MediaSource/readyState}} attribute of the [=parent media source=] to {{ReadyState/""open""}}
   2. [=Queue a task=] to [=fire an event=] named {{sourceopen}} at the [=parent media source=].
6. Run the [=coded frame eviction=] algorithm.

7. If the {{SourceBuffer/[[buffer full flag]]}} equals true, then throw a {{QuotaExceededError}} exception and abort these steps.

   This is the signal that the implementation was unable to evict enough data to accommodate the append or the append is too big. The web application SHOULD use {{SourceBuffer/remove()}} to explicitly free up space and/or reduce the size of the append.

Buffer Append

When {{SourceBuffer/appendBuffer()}} is called, the following steps are run to process the appended data.

1. Run the [=segment parser loop=] algorithm.
2. If the [=segment parser loop=] algorithm in the previous step was aborted, then abort this algorithm.
3. Set the {{SourceBuffer/updating}} attribute to false.
4. [=Queue a task=] to [=fire an event=] named {{update}} at this SourceBuffer object.
5. [=Queue a task=] to [=fire an event=] named {{updateend}} at this SourceBuffer object.

Range Removal

Follow these steps when a caller needs to initiate a JavaScript visible range removal operation that blocks other SourceBuffer updates:

1. Let |start:double| equal the starting [=presentation timestamp=] for the removal range, in seconds measured from [=presentation start time=].
2. Let |end:unrestricted double| equal the end [=presentation timestamp=] for the removal range, in seconds measured from [=presentation start time=].
3. Set the {{SourceBuffer/updating}} attribute to true.
4. [=Queue a task=] to [=fire an event=] named {{updatestart}} at this SourceBuffer object.
5. Return control to the caller and run the rest of the steps asynchronously.
6. Run the [=coded frame removal=] algorithm with |start| and |end| as the start and end of the removal range.
7. Set the {{SourceBuffer/updating}} attribute to false.
8. [=Queue a task=] to [=fire an event=] named {{update}} at this SourceBuffer object.
9. [=Queue a task=] to [=fire an event=] named {{updateend}} at this SourceBuffer object.

Initialization Segment Received

The following steps are run when the [=segment parser loop=] successfully parses a complete [=initialization segment=]:

Each SourceBuffer object has a [[\first initialization segment received flag]] internal slot that tracks whether the first [=initialization segment=] has been appended and received by this algorithm. This flag is set to false when the SourceBuffer is created and updated by the algorithm below.

Each SourceBuffer object has a [[\pending initialization segment for changeType flag]] internal slot that tracks whether an [=initialization segment=] is needed since the most recent {{SourceBuffer/changeType()}}. This flag is set to false when the SourceBuffer is created, set to true by {{SourceBuffer/changeType()}} and reset to false by the algorithm below.

1. Update the {{MediaSource/duration}} attribute if it currently equals NaN:

   If the initialization segment contains a duration:

   Run the [=duration change=] algorithm with |new duration:unrestricted double| set to the duration in the initialization segment.

   Otherwise:

   Run the [=duration change=] algorithm with |new duration| set to positive Infinity.

2. If the [=initialization segment=] has no audio, video, or text tracks, then run the [=append error=] algorithm and abort these steps.
3. If the {{SourceBuffer/[[first initialization segment received flag]]}} is true, then run the following steps:
   1. Verify the following properties. If any of the checks fail then run the [=append error=] algorithm and abort these steps.
      • The number of audio, video, and text tracks match what was in the first [=initialization segment=].
      • If more than one track for a single type are present (e.g., 2 audio tracks), then the [=Track IDs=] match the ones in the first [=initialization segment=].
      • The codecs for each track are supported by the user agent.

        User agents MAY consider codecs, that would otherwise be supported, as "not supported" here if the codecs were not specified in |type:DOMString| parameter passed to (a) the most recently successful {{SourceBuffer/changeType()}} on this {{SourceBuffer}} object, or (b) if no successful {{SourceBuffer/changeType()}} has yet occurred on this object, the {{MediaSource/addSourceBuffer()}} that created this {{SourceBuffer}} object. For example, if the most recently successful {{SourceBuffer/changeType()}} was called with 'video/webm' or 'video/webm; codecs="vp8"', and a video track containing vp9 appears in the initialization segment, then the user agent MAY use this step to trigger a decode error even if the other two properties' checks, above, pass. Implementations are encouraged to trigger error in such cases only when the codec is indeed not supported or the other two properties' checks fail. Web authors are encouraged to use {{SourceBuffer/changeType()}}, {{MediaSource/addSourceBuffer()}} and {{MediaSource/isTypeSupported()}} with precise codec parameters to more proactively detect user agent support. {{SourceBuffer/changeType()}} is required if the {{SourceBuffer}} object's bytestream format is changing.

   2. Add the appropriate [=track descriptions=] from this [=initialization segment=] to each of the [=track buffers=].
   3. Set the [=need random access point flag=] on all track buffers to true.
4. Let |active track flag:boolean| equal false.

5. If the {{SourceBuffer/[[first initialization segment received flag]]}} is false, then run the following steps:

   1. If the [=initialization segment=] contains tracks with codecs the user agent does not support, then run the [=append error=] algorithm and abort these steps.

      User agents MAY consider codecs, that would otherwise be supported, as "not supported" here if the codecs were not specified in |type:DOMString| parameter passed to (a) the most recently successful {{SourceBuffer/changeType()}} on this {{SourceBuffer}} object, or (b) if no successful {{SourceBuffer/changeType()}} has yet occurred on this object, the {{MediaSource/addSourceBuffer()}} that created this {{SourceBuffer}} object. For example, MediaSource.isTypeSupported('video/webm;codecs="vp8,vorbis"') may return true, but if {{MediaSource/addSourceBuffer()}} was called with 'video/webm;codecs="vp8"' and a Vorbis track appears in the [=initialization segment=], then the user agent MAY use this step to trigger a decode error. Implementations are encouraged to trigger error in such cases only when the codec is indeed not supported. Web authors are encouraged to use {{SourceBuffer/changeType()}}, {{MediaSource/addSourceBuffer()}} and {{MediaSource/isTypeSupported()}} with precise codec parameters to more proactively detect user agent support. {{SourceBuffer/changeType()}} is required if the {{SourceBuffer}} object's bytestream format is changing.

   2. For each audio track in the [=initialization segment=], run following steps:

      1. Let |audio byte stream track ID| be the [=Track ID=] for the current track being processed.
      2. Let |audio language:DOMString| be a BCP 47 language tag for the language specified in the [=initialization segment=] for this track or an empty string if no language info is present.
      3. If |audio language| equals the 'und' BCP 47 value, then assign an empty string to |audio language|.
      4. Let |audio label:DOMString| be a label specified in the [=initialization segment=] for this track or an empty string if no label info is present.
      5. Let |audio kinds:DOMString sequence| be a sequence of kind strings specified in the [=initialization segment=] for this track or a sequence with a single empty string element in it if no kind information is provided.
      6. For each value in |audio kinds|, run the following steps:
         1. Let |current audio kind:DOMString| equal the value from |audio kinds| for this iteration of the loop.
         2. Let |new audio track:AudioTrack| be a new {{AudioTrack}} object.
         3. Generate a unique ID and assign it to the {{AudioTrack/id}} property on |new audio track|.
         4. Assign |audio language| to the {{AudioTrack/language}} property on |new audio track|.
         5. Assign |audio label| to the {{AudioTrack/label}} property on |new audio track|.
         6. Assign |current audio kind| to the {{AudioTrack/kind}} property on |new audio track|.

         7. If this {{SourceBuffer}} object's {{SourceBuffer/audioTracks}}.{{AudioTrackList/length}} equals 0, then run the following steps:

            1. Set the {{AudioTrack/enabled}} property on |new audio track| to true.
            2. Set |active track flag| to true.
         8. Add |new audio track| to the {{SourceBuffer/audioTracks}} attribute on this SourceBuffer object.

            This should trigger {{AudioTrackList}} [[HTML]] logic to [=queue a task=] to [=fire an event=] named [=AudioTrackList/addtrack=] using {{TrackEvent}} with the {{TrackEvent/track}} attribute initialized to |new audio track|, at the {{AudioTrackList}} object referenced by the {{SourceBuffer/audioTracks}} attribute on this SourceBuffer object.

         9. If the [=parent media source=] was constructed in a {{DedicatedWorkerGlobalScope}}:

            Post an internal create track mirror message to {{MediaSource/[[port to main]]}} whose implicit handler in {{Window}} runs the following steps:

            1. Let |mirrored audio track:AudioTrack| be a new {{AudioTrack}} object.
            2. Assign the same property values to |mirrored audio track| as were determined for |new audio track|.
            3. Add |mirrored audio track| to the {{HTMLMediaElement/audioTracks}} attribute on the HTMLMediaElement.

            Otherwise:

            Add |new audio track| to the {{HTMLMediaElement/audioTracks}} attribute on the HTMLMediaElement.

            This should trigger {{AudioTrackList}} [[HTML]] logic to [=queue a task=] to [=fire an event=] named [=AudioTrackList/addtrack=] using {{TrackEvent}} with the {{TrackEvent/track}} attribute initialized to |mirrored audio track| or |new audio track|, at the {{AudioTrackList}} object referenced by the {{HTMLMediaElement/audioTracks}} attribute on the HTMLMediaElement.

      7. Create a new [=track buffer=] to store [=coded frames=] for this track.
      8. Add the [=track description=] for this track to the [=track buffer=].

   3. For each video track in the [=initialization segment=], run following steps:

      1. Let |video byte stream track ID| be the [=Track ID=] for the current track being processed.
      2. Let |video language:DOMString| be a BCP 47 language tag for the language specified in the [=initialization segment=] for this track or an empty string if no language info is present.
      3. If |video language| equals the 'und' BCP 47 value, then assign an empty string to |video language|.
      4. Let |video label:DOMString| be a label specified in the [=initialization segment=] for this track or an empty string if no label info is present.
      5. Let |video kinds:DOMString sequence| be a sequence of kind strings specified in the [=initialization segment=] for this track or a sequence with a single empty string element in it if no kind information is provided.
      6. For each value in |video kinds|, run the following steps:
         1. Let |current video kind:DOMString| equal the value from |video kinds| for this iteration of the loop.
         2. Let |new video track:VideoTrack| be a new {{VideoTrack}} object.
         3. Generate a unique ID and assign it to the {{VideoTrack/id}} property on |new video track|.
         4. Assign |video language| to the {{VideoTrack/language}} property on |new video track|.
         5. Assign |video label| to the {{VideoTrack/label}} property on |new video track|.
         6. Assign |current video kind| to the {{VideoTrack/kind}} property on |new video track|.

         7. If this {{SourceBuffer}} object's {{SourceBuffer/videoTracks}}.{{VideoTrackList/length}} equals 0, then run the following steps:

            1. Set the {{VideoTrack/selected}} property on |new video track| to true.
            2. Set |active track flag| to true.
         8. Add |new video track| to the {{SourceBuffer/videoTracks}} attribute on this SourceBuffer object.

            This should trigger {{VideoTrackList}} [[HTML]] logic to [=queue a task=] to [=fire an event=] named [=VideoTrackList/addtrack=] using {{TrackEvent}} with the {{TrackEvent/track}} attribute initialized to |new video track|, at the {{VideoTrackList}} object referenced by the {{SourceBuffer/videoTracks}} attribute on this SourceBuffer object.

         9. If the [=parent media source=] was constructed in a {{DedicatedWorkerGlobalScope}}:

            Post an internal create track mirror message to {{MediaSource/[[port to main]]}} whose implicit handler in {{Window}} runs the following steps:

            1. Let |mirrored video track:VideoTrack| be a new {{VideoTrack}} object.
            2. Assign the same property values to |mirrored video track| as were determined for |new video track|.
            3. Add |mirrored video track| to the {{HTMLMediaElement/videoTracks}} attribute on the HTMLMediaElement.

            Otherwise:

            Add |new video track| to the {{HTMLMediaElement/videoTracks}} attribute on the HTMLMediaElement.

            This should trigger {{VideoTrackList}} [[HTML]] logic to [=queue a task=] to [=fire an event=] named [=VideoTrackList/addtrack=] using {{TrackEvent}} with the {{TrackEvent/track}} attribute initialized to |mirrored video track| or |new video track|, at the {{VideoTrackList}} object referenced by the {{HTMLMediaElement/videoTracks}} attribute on the HTMLMediaElement.

      7. Create a new [=track buffer=] to store [=coded frames=] for this track.
      8. Add the [=track description=] for this track to the [=track buffer=].

   4. For each text track in the [=initialization segment=], run following steps:

      1. Let |text byte stream track ID| be the [=Track ID=] for the current track being processed.
      2. Let |text language:DOMString| be a BCP 47 language tag for the language specified in the [=initialization segment=] for this track or an empty string if no language info is present.
      3. If |text language| equals the 'und' BCP 47 value, then assign an empty string to |text language|.
      4. Let |text label:DOMString| be a label specified in the [=initialization segment=] for this track or an empty string if no label info is present.
      5. Let |text kinds:DOMString sequence| be a sequence of kind strings specified in the [=initialization segment=] for this track or a sequence with a single empty string element in it if no kind information is provided.
      6. For each value in |text kinds|, run the following steps:
         1. Let |current text kind:DOMString| equal the value from |text kinds| for this iteration of the loop.
         2. Let |new text track:TextTrack| be a new {{TextTrack}} object.
         3. Generate a unique ID and assign it to the {{TextTrack/id}} property on |new text track|.
         4. Assign |text language| to the {{TextTrack/language}} property on |new text track|.
         5. Assign |text label| to the {{TextTrack/label}} property on |new text track|.
         6. Assign |current text kind| to the {{TextTrack/kind}} property on |new text track|.
         7. Populate the remaining properties on |new text track| with the appropriate information from the [=initialization segment=].
         8. If the {{TextTrack/mode}} property on |new text track| equals or , then set |active track flag| to true.
         9. Add |new text track| to the {{SourceBuffer/textTracks}} attribute on this SourceBuffer object.

            This should trigger {{TextTrackList}} [[HTML]] logic to [=queue a task=] to [=fire an event=] named [=TextTrackList/addtrack=] using {{TrackEvent}} with the {{TrackEvent/track}} attribute initialized to |new text track|, at the {{TextTrackList}} object referenced by the {{SourceBuffer/textTracks}} attribute on this SourceBuffer object.

         10. If the [=parent media source=] was constructed in a {{DedicatedWorkerGlobalScope}}:

             Post an internal create track mirror message to {{MediaSource/[[port to main]]}} whose implicit handler in {{Window}} runs the following steps:

             1. Let |mirrored text track:TextTrack| be a new {{TextTrack}} object.
             2. Assign the same property values to |mirrored text track| as were determined for |new text track|.
             3. Add |mirrored text track| to the {{HTMLMediaElement/textTracks}} attribute on the HTMLMediaElement.

             Otherwise:

             Add |new text track| to the {{HTMLMediaElement/textTracks}} attribute on the HTMLMediaElement.

             This should trigger {{TextTrackList}} [[HTML]] logic to [=queue a task=] to [=fire an event=] named [=TextTrackList/addtrack=] using {{TrackEvent}} with the {{TrackEvent/track}} attribute initialized to |mirrored text track| or |new text track|, at the {{TextTrackList}} object referenced by the {{HTMLMediaElement/textTracks}} attribute on the HTMLMediaElement.

      7. Create a new [=track buffer=] to store [=coded frames=] for this track.
      8. Add the [=track description=] for this track to the [=track buffer=].
   5. If |active track flag| equals true, then run the following steps:
      1. Add this SourceBuffer to {{MediaSource/activeSourceBuffers}}.
      2. [=Queue a task=] to [=fire an event=] named {{addsourcebuffer}} at {{MediaSource/activeSourceBuffers}}
   6. Set {{SourceBuffer/[[first initialization segment received flag]]}} to true.
6. Set {{SourceBuffer/[[pending initialization segment for changeType flag]]}} to false.
7. If the |active track flag| equals true, then run the following steps:
8. Use the [=parent media source=]'s [=mirror if necessary=] algorithm to run the following step in {{Window}}:
   1. If the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute is greater than {{HTMLMediaElement/HAVE_CURRENT_DATA}}, then set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to {{HTMLMediaElement/HAVE_METADATA}}.

      Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement.

9. If each object in {{MediaSource/sourceBuffers}} of the [=parent media source=] has {{SourceBuffer/[[first initialization segment received flag]]}} equal to true, then use the [=parent media source=]'s [=mirror if necessary=] algorithm to run the following step in {{Window}}:
   1. If the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute is {{HTMLMediaElement/HAVE_NOTHING}}, then set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to {{HTMLMediaElement/HAVE_METADATA}}.

      Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement. If transition from {{HTMLMediaElement/HAVE_NOTHING}} to {{HTMLMediaElement/HAVE_METADATA}} occurs, it should trigger HTMLMediaElement logic to [=queue a task=] to [=fire an event=] named [=HTMLMediaElement/loadedmetadata=] at the media element.

Coded Frame Processing

When complete [=coded frames=] have been parsed by the [=segment parser loop=] then the following steps are run:

1. For each [=coded frame=] in the [=media segment=] run the following steps:

   1. Loop Top:

      If {{SourceBuffer/[[generate timestamps flag]]}} equals true:

      1. Let |presentation timestamp:double| equal 0.
      2. Let |decode timestamp:double| equal 0.

      Otherwise:

      1. Let |presentation timestamp| be a double precision floating point representation of the coded frame's [=presentation timestamp=] in seconds.

         Special processing may be needed to determine the presentation and decode timestamps for timed text frames since this information may not be explicitly present in the underlying format or may be dependent on the order of the frames. Some metadata text tracks, like MPEG2-TS PSI data, may only have implied timestamps. Format specific rules for these situations SHOULD be in the [=byte stream format specifications=] or in separate extension specifications.

      2. Let |decode timestamp| be a double precision floating point representation of the coded frame's decode timestamp in seconds.

         Implementations don't have to internally store timestamps in a double precision floating point representation. This representation is used here because it is the representation for timestamps in the HTML spec. The intention here is to make the behavior clear without adding unnecessary complexity to the algorithm to deal with the fact that adding a timestampOffset may cause a timestamp rollover in the underlying timestamp representation used by the byte stream format. Implementations can use any internal timestamp representation they wish, but the addition of timestampOffset SHOULD behave in a similar manner to what would happen if a double precision floating point representation was used.

   2. Let |frame duration:double| be a double precision floating point representation of the [=coded frame duration|coded frame's duration=] in seconds.
   3. If {{SourceBuffer/mode}} equals {{AppendMode/""sequence""}} and {{SourceBuffer/[[group start timestamp]]}} is set, then run the following steps:
      1. Set {{SourceBuffer/timestampOffset}} equal to {{SourceBuffer/[[group start timestamp]]}} minus |presentation timestamp|.
      2. Set {{SourceBuffer/[[group end timestamp]]}} equal to {{SourceBuffer/[[group start timestamp]]}}.
      3. Set the [=need random access point flag=] on all [=track buffers=] to true.
      4. Unset {{SourceBuffer/[[group start timestamp]]}}.

   4. If {{SourceBuffer/timestampOffset}} is not 0, then run the following steps:

      1. Add {{SourceBuffer/timestampOffset}} to the |presentation timestamp|.
      2. Add {{SourceBuffer/timestampOffset}} to the |decode timestamp|.
   5. Let |track buffer| equal the [=track buffer=] that the coded frame will be added to.

   6. If [=last decode timestamp=] for |track buffer| is set and |decode timestamp| is less than [=last decode timestamp=]:

      OR

      If [=last decode timestamp=] for |track buffer| is set and the difference between |decode timestamp| and [=last decode timestamp=] is greater than 2 times [=last frame duration=]:

      1. If {{SourceBuffer/mode}} equals {{AppendMode/""segments""}}:

         Set {{SourceBuffer/[[group end timestamp]]}} to |presentation timestamp|.

         If {{SourceBuffer/mode}} equals {{AppendMode/""sequence""}}:

         Set {{SourceBuffer/[[group start timestamp]]}} equal to the {{SourceBuffer/[[group end timestamp]]}}.

      2. Unset the [=last decode timestamp=] on all [=track buffers=].
      3. Unset the [=last frame duration=] on all [=track buffers=].
      4. Unset the [=highest end timestamp=] on all [=track buffers=].
      5. Set the [=need random access point flag=] on all [=track buffers=] to true.
      6. Jump to the Loop Top step above to restart processing of the current [=coded frame=].

      Otherwise:

      Continue.

   7. Let |frame end timestamp:double| equal the sum of |presentation timestamp| and |frame duration|.
   8. If |presentation timestamp| is less than {{SourceBuffer/appendWindowStart}}, then set the [=need random access point flag=] to true, drop the coded frame, and jump to the top of the loop to start processing the next coded frame.

      Some implementations MAY choose to collect some of these coded frames with |presentation timestamp| less than {{SourceBuffer/appendWindowStart}} and use them to generate a splice at the first coded frame that has a [=presentation timestamp=] greater than or equal to {{SourceBuffer/appendWindowStart}} even if that frame is not a [=random access point=]. Supporting this requires multiple decoders or faster than real-time decoding so for now this behavior will not be a normative requirement.

   9. If |frame end timestamp| is greater than {{SourceBuffer/appendWindowEnd}}, then set the [=need random access point flag=] to true, drop the coded frame, and jump to the top of the loop to start processing the next coded frame.

      Some implementations MAY choose to collect coded frames with |presentation timestamp| less than {{SourceBuffer/appendWindowEnd}} and |frame end timestamp| greater than {{SourceBuffer/appendWindowEnd}} and use them to generate a splice across the portion of the collected coded frames within the append window at time of collection, and the beginning portion of later processed frames which only partially overlap the end of the collected coded frames. Supporting this requires multiple decoders or faster than real-time decoding so for now this behavior will not be a normative requirement. In conjunction with collecting coded frames that span {{SourceBuffer/appendWindowStart}}, implementations MAY thus support gapless audio splicing.

   10. If the [=need random access point flag=] on |track buffer| equals true, then run the following steps:
       1. If the coded frame is not a [=random access point=], then drop the coded frame and jump to the top of the loop to start processing the next coded frame.
       2. Set the [=need random access point flag=] on |track buffer| to false.
   11. Let |spliced audio frame| be an unset variable for holding audio splice information
   12. Let |spliced timed text frame| be an unset variable for holding timed text splice information
   13. If [=last decode timestamp=] for |track buffer| is unset and |presentation timestamp| falls within the [=presentation interval=] of a [=coded frame=] in |track buffer|, then run the following steps:
       1. Let |overlapped frame| be the [=coded frame=] in |track buffer| that matches the condition above.

       2. If |track buffer| contains audio [=coded frames=]:

          Run the [=audio splice frame=] algorithm and if a splice frame is returned, assign it to |spliced audio frame|.

          If |track buffer| contains video [=coded frames=]:

          1. Let |remove window timestamp:double| equal the |overlapped frame| [=presentation timestamp=] plus 1 microsecond.
          2. If the |presentation timestamp| is less than the |remove window timestamp|, then remove |overlapped frame| from |track buffer|.

             This is to compensate for minor errors in frame timestamp computations that can appear when converting back and forth between double precision floating point numbers and rationals. This tolerance allows a frame to replace an existing one as long as it is within 1 microsecond of the existing frame's start time. Frames that come slightly before an existing frame are handled by the removal step below.

          If |track buffer| contains timed text [=coded frames=]:

          Run the [=text splice frame=] algorithm and if a splice frame is returned, assign it to |spliced timed text frame|.

   14. Remove existing coded frames in |track buffer|:

       If [=highest end timestamp=] for |track buffer| is not set:

       Remove all [=coded frames=] from |track buffer| that have a [=presentation timestamp=] greater than or equal to |presentation timestamp| and less than |frame end timestamp|.

       If [=highest end timestamp=] for |track buffer| is set and less than or equal to |presentation timestamp|:

       Remove all [=coded frames=] from |track buffer| that have a [=presentation timestamp=] greater than or equal to [=highest end timestamp=] and less than |frame end timestamp|.

   15. Remove all possible decoding dependencies on the [=coded frames=] removed in the previous two steps by removing all [=coded frames=] from |track buffer| between those frames removed in the previous two steps and the next [=random access point=] after those removed frames.

       Removing all [=coded frames=] until the next [=random access point=] is a conservative estimate of the decoding dependencies since it assumes all frames between the removed frames and the next random access point depended on the frames that were removed.

   16. If |spliced audio frame| is set:

       Add |spliced audio frame| to the |track buffer|.

       If |spliced timed text frame| is set:

       Add |spliced timed text frame| to the |track buffer|.

       Otherwise:

       Add the [=coded frame=] with the |presentation timestamp|, |decode timestamp|, and |frame duration| to the |track buffer|.

   17. Set [=last decode timestamp=] for |track buffer| to |decode timestamp|.
   18. Set [=last frame duration=] for |track buffer| to |frame duration|.
   19. If [=highest end timestamp=] for |track buffer| is unset or |frame end timestamp| is greater than [=highest end timestamp=], then set [=highest end timestamp=] for |track buffer| to |frame end timestamp|.

       The greater than check is needed because bidirectional prediction between coded frames can cause |presentation timestamp| to not be monotonically increasing even though the decode timestamps are monotonically increasing.

   20. If |frame end timestamp| is greater than {{SourceBuffer/[[group end timestamp]]}}, then set {{SourceBuffer/[[group end timestamp]]}} equal to |frame end timestamp|.
   21. If {{SourceBuffer/[[generate timestamps flag]]}} equals true, then set {{SourceBuffer/timestampOffset}} equal to |frame end timestamp|.

2. If the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute is {{HTMLMediaElement/HAVE_METADATA}} and the new [=coded frames=] cause {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}} to have a {{TimeRanges}} for the current playback position, then set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to {{HTMLMediaElement/HAVE_CURRENT_DATA}}.

   Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement.

3. If the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute is {{HTMLMediaElement/HAVE_CURRENT_DATA}} and the new [=coded frames=] cause {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}} to have a {{TimeRanges}} that includes the current playback position and some time beyond the current playback position, then set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to {{HTMLMediaElement/HAVE_FUTURE_DATA}}.

   Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement.

4. If the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute is {{HTMLMediaElement/HAVE_FUTURE_DATA}} and the new [=coded frames=] cause {{HTMLMediaElement}}.{{HTMLMediaElement/buffered}} to have a {{TimeRanges}} that includes the current playback position and [=enough data to ensure uninterrupted playback=], then set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to {{HTMLMediaElement/HAVE_ENOUGH_DATA}}.

   Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement.

5. If the [=media segment=] contains data beyond the current {{MediaSource/duration}}, then run the [=duration change=] algorithm with |new duration:unrestricted double| set to the maximum of the current duration and the {{SourceBuffer/[[group end timestamp]]}}.

Coded Frame Removal

Follow these steps when [=coded frames=] for a specific time range need to be removed from the SourceBuffer:

1. Let |start:double| be the starting [=presentation timestamp=] for the removal range.
2. Let |end:unrestricted double| be the end [=presentation timestamp=] for the removal range.

3. For each [=track buffer=] in this {{SourceBuffer}}, run the following steps:

   1. Let |remove end timestamp:unrestricted double| be the current value of {{MediaSource/duration}}

   2. If this [=track buffer=] has a [=random access point=] timestamp that is greater than or equal to |end|, then update |remove end timestamp| to that random access point timestamp.

      Random access point timestamps can be different across tracks because the dependencies between [=coded frames=] within a track are usually different than the dependencies in another track.

   3. Remove all media data, from this [=track buffer=], that contain starting timestamps greater than or equal to |start| and less than the |remove end timestamp|.

      1. For each removed frame, if the frame has a [=decode timestamp=] equal to the [=last decode timestamp=] for the frame's track, run the following steps:

         If {{SourceBuffer/mode}} equals {{AppendMode/""segments""}}:

         Set {{SourceBuffer/[[group end timestamp]]}} to [=presentation timestamp=].

         If {{SourceBuffer/mode}} equals {{AppendMode/""sequence""}}:

         Set {{SourceBuffer/[[group start timestamp]]}} equal to the {{SourceBuffer/[[group end timestamp]]}}.

      2. Unset the [=last decode timestamp=] on all [=track buffers=].
      3. Unset the [=last frame duration=] on all [=track buffers=].
      4. Unset the [=highest end timestamp=] on all [=track buffers=].
      5. Set the [=need random access point flag=] on all [=track buffers=] to true.
   4. Remove all possible decoding dependencies on the [=coded frames=] removed in the previous step by removing all [=coded frames=] from this [=track buffer=] between those frames removed in the previous step and the next [=random access point=] after those removed frames.

      Removing all [=coded frames=] until the next [=random access point=] is a conservative estimate of the decoding dependencies since it assumes all frames between the removed frames and the next random access point depended on the frames that were removed.

   5. If this object is in {{MediaSource/activeSourceBuffers}}, the is greater than or equal to |start| and less than the |remove end timestamp|, and {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} is greater than {{HTMLMediaElement/HAVE_METADATA}}, then set the {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} attribute to {{HTMLMediaElement/HAVE_METADATA}} and stall playback.

      Per [[HTML]] logic, {{HTMLMediaElement}}.{{HTMLMediaElement/readyState}} changes may trigger events on the HTMLMediaElement.

      This transition occurs because media data for the current position has been removed. Playback cannot progress until media for the is appended or the selected/enabled tracks change.

4. If the {{SourceBuffer/[[buffer full flag]]}} equals true and this object is ready to accept more bytes, then set the {{SourceBuffer/[[buffer full flag]]}} to false.

Coded Frame Eviction

This algorithm is run to free up space in this {{SourceBuffer}} when new data is appended.

1. Let |new data:BufferSource| equal the data that is about to be appended to this SourceBuffer.

   Need to recognize step here that implementations MAY decide to set {{SourceBuffer/[[buffer full flag]]}} true here if it predicts that processing |new data| in addition to any existing bytes in {{SourceBuffer/[[input buffer]]}} would exceed the capacity of the {{SourceBuffer}}. Such a step enables more proactive push-back from implementations before accepting |new data| which would overflow resources, for example. In practice, at least one implementation already does this.

2. If the {{SourceBuffer/[[buffer full flag]]}} equals false, then abort these steps.
3. Let |removal ranges:normalized TimeRanges| equal a list of presentation time ranges that can be evicted from the presentation to make room for the |new data|.

   Implementations MAY use different methods for selecting |removal ranges| so web applications SHOULD NOT depend on a specific behavior. The web application can use the {{SourceBuffer/buffered}} attribute to observe whether portions of the buffered data have been evicted.

4. For each range in |removal ranges|, run the [=coded frame removal=] algorithm with |start:double| and |end:unrestricted double| equal to the removal range start and end timestamp respectively.

Audio Splice Frame

Follow these steps when the [=coded frame processing=] algorithm needs to generate a splice frame for two overlapping audio [=coded frames=]:

1. Let |track buffer| be the [=track buffer=] that will contain the splice.
2. Let |new coded frame| be the new [=coded frame=], that is being added to |track buffer|, which triggered the need for a splice.
3. Let |presentation timestamp:double| be the [=presentation timestamp=] for |new coded frame|.
4. Let |decode timestamp:double| be the decode timestamp for |new coded frame|.
5. Let |frame duration:double| be the [=coded frame duration=] of |new coded frame|.
6. Let |overlapped frame| be the [=coded frame=] in |track buffer| with a [=presentation interval=] that contains |presentation timestamp|.
7. Update |presentation timestamp| and |decode timestamp| to the nearest audio sample timestamp based on sample rate of the audio in |overlapped frame|. If a timestamp is equidistant from both audio sample timestamps, then use the higher timestamp (e.g., floor(x * sample_rate + 0.5) / sample_rate).

   For example, given the following values:

   • The [=presentation timestamp=] of |overlapped frame| equals 10.
   • The sample rate of |overlapped frame| equals 8000 Hz
   • |presentation timestamp| equals 10.01255
   • |decode timestamp| equals 10.01255

   |presentation timestamp| and |decode timestamp| are updated to 10.0125 since 10.01255 is closer to 10 + 100/8000 (10.0125) than 10 + 101/8000 (10.012625)

8. If the user agent does not support crossfading then run the following steps:
   1. Remove |overlapped frame| from |track buffer|.
   2. Add a silence frame to |track buffer| with the following properties:
      • The [=presentation timestamp=] set to the |overlapped frame| [=presentation timestamp=].
      • The [=decode timestamp=] set to the |overlapped frame| [=decode timestamp=].
      • The [=coded frame duration=] set to difference between |presentation timestamp| and the |overlapped frame| [=presentation timestamp=].

      Some implementations MAY apply fades to/from silence to coded frames on either side of the inserted silence to make the transition less jarring.

   3. Return to caller without providing a splice frame.

      This is intended to allow |new coded frame| to be added to the |track buffer| as if |overlapped frame| had not been in the |track buffer| to begin with.

9. Let |frame end timestamp:double| equal the sum of |presentation timestamp| and |frame duration|.
10. Let |splice end timestamp:double| equal the sum of |presentation timestamp| and the splice duration of 5 milliseconds.
11. Let |fade out coded frames| equal |overlapped frame| as well as any additional frames in |track buffer| that have a [=presentation timestamp=] greater than |presentation timestamp| and less than |splice end timestamp|.
12. Remove all the frames included in |fade out coded frames| from |track buffer|.
13. Return a splice frame with the following properties:
    • The [=presentation timestamp=] set to the |overlapped frame| [=presentation timestamp=].
    • The [=decode timestamp=] set to the |overlapped frame| [=decode timestamp=].
    • The [=coded frame duration=] set to difference between |frame end timestamp| and the |overlapped frame| [=presentation timestamp=].
    • The fade out coded frames equals |fade out coded frames|.
    • The fade in coded frame equals |new coded frame|.

      If the |new coded frame| is less than 5 milliseconds in duration, then coded frames that are appended after the |new coded frame| will be needed to properly render the splice.

    • The splice timestamp equals |presentation timestamp|.

    See the [=audio splice rendering=] algorithm for details on how this splice frame is rendered.

Audio Splice Rendering

The following steps are run when a spliced frame, generated by the [=audio splice frame=] algorithm, needs to be rendered by the media element:

1. Let |fade out coded frames| be the [=coded frames=] that are faded out during the splice.
2. Let |fade in coded frames| be the [=coded frames=] that are faded in during the splice.
3. Let |presentation timestamp:double| be the [=presentation timestamp=] of the first coded frame in |fade out coded frames|.
4. Let |end timestamp:double| be the sum of the [=presentation timestamp=] and the [=coded frame duration=] of the last frame in |fade in coded frames|.
5. Let |splice timestamp:double| be the [=presentation timestamp=] where the splice starts. This corresponds with the [=presentation timestamp=] of the first frame in |fade in coded frames|.
6. Let |splice end timestamp:double| equal |splice timestamp| plus five milliseconds.
7. Let |fade out samples| be the samples generated by decoding |fade out coded frames|.
8. Trim |fade out samples| so that it only contains samples between |presentation timestamp| and |splice end timestamp|.
9. Let |fade in samples| be the samples generated by decoding |fade in coded frames|.
10. If |fade out samples| and |fade in samples| do not have a common sample rate and channel layout, then convert |fade out samples| and |fade in samples| to a common sample rate and channel layout.
11. Let |output samples| be a buffer to hold the output samples.
12. Apply a linear gain fade out with a starting gain of 1 and an ending gain of 0 to the samples between |splice timestamp| and |splice end timestamp| in |fade out samples|.
13. Apply a linear gain fade in with a starting gain of 0 and an ending gain of 1 to the samples between |splice timestamp| and |splice end timestamp| in |fade in samples|.
14. Copy samples between |presentation timestamp| to |splice timestamp| from |fade out samples| into |output samples|.
15. For each sample between |splice timestamp| and |splice end timestamp|, compute the sum of a sample from |fade out samples| and the corresponding sample in |fade in samples| and store the result in |output samples|.
16. Copy samples between |splice end timestamp| to |end timestamp| from |fade in samples| into |output samples|.
17. Render |output samples|.

Here is a graphical representation of this algorithm.

Text Splice Frame

Follow these steps when the [=coded frame processing=] algorithm needs to generate a splice frame for two overlapping timed text [=coded frames=]:

1. Let |track buffer| be the [=track buffer=] that will contain the splice.
2. Let |new coded frame| be the new [=coded frame=], that is being added to |track buffer|, which triggered the need for a splice.
3. Let |presentation timestamp:double| be the [=presentation timestamp=] for |new coded frame|
4. Let |decode timestamp:double| be the decode timestamp for |new coded frame|.
5. Let |frame duration:double| be the [=coded frame duration=] of |new coded frame|.
6. Let |frame end timestamp:double| equal the sum of |presentation timestamp| and |frame duration|.
7. Let |first overlapped frame| be the [=coded frame=] in |track buffer| with a [=presentation interval=] that contains |presentation timestamp|.
8. Let |overlapped presentation timestamp:double| be the [=presentation timestamp=] of the |first overlapped frame|.
9. Let |overlapped frames| equal |first overlapped frame| as well as any additional frames in |track buffer| that have a [=presentation timestamp=] greater than |presentation timestamp| and less than |frame end timestamp|.
10. Remove all the frames included in |overlapped frames| from |track buffer|.
11. Update the [=coded frame duration=] of the |first overlapped frame| to |presentation timestamp| minus |overlapped presentation timestamp|.
12. Add |first overlapped frame| to the |track buffer|.
13. Return to caller without providing a splice frame.

    This is intended to allow |new coded frame| to be added to the |track buffer| as if it hadn't overlapped any frames in |track buffer| to begin with.