Fixes#3993 - WebRTC streams recorded to MP4 via DVR exhibit audio/video
synchronization issues, with audio typically ahead of video. **Note:
This issue is specific to MP4 format; FLV recordings are not affected.**
When WebRTC streams are converted to RTMP and then muxed to MP4, the
audio and video tracks may start at different timestamps. The MP4 muxer
was not accounting for this timing offset between the first audio and
video samples in the STTS (Sample Time-to-Sample) table, causing the
tracks to be misaligned in the final MP4 file.
Introduces `SrsMp4DvrJitter` class specifically for MP4 audio/video
synchronization:
- **Timestamp Tracking**: Records the DTS of the first audio and video
samples
- **Offset Calculation**: Computes the timing difference between track
start times
- **MP4 STTS Correction**: Sets appropriate `sample_delta` values in the
MP4 STTS table to maintain proper A/V sync
- Added `SrsMp4DvrJitter` class in `srs_kernel_mp4.hpp/cpp`
- Integrated jitter correction into `SrsMp4SampleManager::write_track()`
for MP4 format only
- Added comprehensive unit tests covering various timing scenarios
- **Scope**: Changes are isolated to MP4 kernel code and do not affect
FLV processing
This fix ensures that MP4 DVR recordings from WebRTC streams maintain
proper audio/video synchronization regardless of the relative timing of
the first audio and video frames, while leaving FLV format processing
unchanged.
---------
Co-authored-by: Haibo Chen <495810242@qq.com>
Co-authored-by: john <hondaxiao@tencent.com>
Co-authored-by: winlin <winlinvip@gmail.com>
Co-authored-by: OSSRS-AI <winlinam@gmail.com>
## Problem
The `valgrind?check=new` API parameter uses `VALGRIND_DO_NEW_LEAK_CHECK`
which is only available in Valgrind 3.21+. On older versions like
CentOS's default Valgrind 3.16, this causes undefined behavior since the
macro is not defined.
## Solution
- Check for `VALGRIND_DO_NEW_LEAK_CHECK` availability before processing
the request
- Return `ERROR_NOT_SUPPORTED` with version information when unsupported
- Move the version check before thread creation to avoid unnecessary
resource allocation
## Changes
- Early validation of `check=new` parameter compatibility
- Proper error response with current Valgrind version details
- Prevents undefined behavior on older Valgrind installations
Fixes compatibility issues with older Valgrind versions commonly found
in enterprise Linux distributions.
---------
Co-authored-by: Jacob Su <suzp1984@gmail.com>
Co-authored-by: winlin <winlinvip@gmail.com>
Co-authored-by: OSSRS-AI <winlinam@gmail.com>
## How to reproduce?
1. cmake version 4.0.3
2. clean srt build cache:
`rm -rf objs/Platform-*`
3. `./configure`
compiling error output:
> Build srt-1-fit
> patching file
'./objs/Platform-SRS7-Darwin-24.6.0-Clang17.0.0-arm64/srt-1-fit/srtcore/api.cpp'
> Running: cmake .
-DCMAKE_INSTALL_PREFIX=/Users/jacobsu/hack/media/srs/trunk/objs/Platform-SRS7-Darwin-24.6.0-Clang17.0.0-arm64/3rdparty/srt
-DENABLE_APPS=0 -DENABLE_STATIC=1 -DENABLE_CXX11=0 -DENABLE_SHARED=0
-DOPENSSL_INCLUDE_DIR=/usr/local/opt/openssl/include
-DOPENSSL_LIBRARIES=/usr/local/opt/openssl/lib/libcrypto.a
> CMake Error at CMakeLists.txt:10 (cmake_minimum_required):
> Compatibility with CMake < 3.5 has been removed from CMake.
>
> Update the VERSION argument <min> value. Or, use the <min>...<max>
syntax
> to tell CMake that the project requires at least <min> but has been
updated
> to work with policies introduced by <max> or earlier.
>
> Or, add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to try configuring anyway.
>
> -- Configuring incomplete, errors occurred!
## Cause
CMake 4.x not long compatible with function cmake_minimum_required
(VERSION 2.8.12 FATAL_ERROR) with only min version anymore.
## Solution
add `add -DCMAKE_POLICY_VERSION_MINIMUM=3.5` to cmake cmd args.
---------
Co-authored-by: Haibo Chen <495810242@qq.com>
Co-authored-by: OSSRS-AI <winlinam@gmail.com>
## Cause
`rtc_server.listen` conflict for conf `origin[1,2,3]-for-proxy.conf`
## How to reproduce?
follow the tutorial
`https://ossrs.net/lts/en-us/docs/v7/doc/origin-cluster`.
The webrtc play not works, when start more than one origin srs server.
Co-authored-by: Winlin <winlinvip@gmail.com>
Currently, SRS only supports HLS with MPEG-TS format segment files, but
for LL-HLS and HEVC, it requires the fMP4 format. See #4327 for details.
Furthermore, fMP4 has a smaller overhead compared to TS, and fMP4 can be
used for DVR. In short, fMP4 is definitely the future segment format for
HLS.
Start SRS with the config file that enables HLS with fMP4:
```
./objs/srs -c conf/hls.mp4.conf
```
Publish stream by FFmpeg:
```
ffmpeg -re -i doc/source.flv -c copy -f flv rtmp://localhost/live/livestream
```
Play the stream by SRS player:
[http://localhost:8080/live/livestream.m3u8](http://localhost:8080/players/srs_player.html?stream=livestream.m3u8)
Finished by AI:
* [AI: Change init.mp4 to the same directory of
m3u8.](17621c8442)
* [AI: Fix the error handling
bug.](af3758a592)
* [AI: Fix Chrome stuttering
problem.](aaab60c314)
---------
Co-authored-by: winlin <winlinvip@gmail.com>
## Summary
Fixes a critical heap-use-after-free crash in HTTP-FLV streaming that
occurs when a client requests a stream while it's being unmounted
asynchronously.
## Problem
- **Issue**: #4429 - Heap-use-after-free crash in
`SrsLiveStream::serve_http()`
- **Root Cause**: Race condition between coroutines in single-threaded
SRS server:
1. **Coroutine A**: HTTP client requests FLV stream → `serve_http()`
starts
2. **Coroutine B**: RTMP publisher disconnects → triggers async stream
destruction
3. **Async Worker**: Destroys `SrsLiveStream` object while Coroutine A
is yielded
4. **Coroutine A**: Resumes and accesses freed memory → **CRASH**
## Solution
1. **Early viewer registration**: Add HTTP connection to `viewers_` list
immediately in `serve_http()` before any I/O operations that could yield
2. **Lifecycle protection**: Split `serve_http()` into wrapper and
implementation to ensure proper viewer management
3. **Stream availability checks**: Add fast checks for stream disposal
state before critical operations
4. **Improved error handling**: Convert warnings to fatal errors when
trying to free alive streams
## Key Changes
- **`SrsLiveStream::serve_http()`**: Now immediately registers viewer
and delegates to `serve_http_impl()`
- **`SrsLiveStream::serve_http_impl()`**: Contains the actual HTTP
serving logic
- **`SrsHttpStreamDestroy::call()`**: Enhanced error handling and longer
wait timeout
- **Stream state validation**: Added checks for `entry->enabled` before
proceeding with stream operations
Fixes#4429
Rtp packets may be retransmitted, disordered, jittery, delayed,
etc.There may be abnormalities when converting to rtmp.
To reproduce this problem, you need to set the network reordering by
[tc-ui](https://github.com/ossrs/tc-ui). Note that you need a linux
server, and start it by docker:
```bash
docker run --network=host --privileged -it --restart always -d \
--name tc -v /lib/modules:/lib/modules:ro ossrs/tc-ui:1
```
Set up 5% packet reordering and a 1ms delay; then you will notice that
the audio is stuttering, somewhat noisy, and lacks fluency.
```bash
curl http://localhost:2023/tc/api/v1/config/raw -X POST \
-d 'tcset ens5 --direction incoming --delay 40ms --reordering 5% --port 8000'
```
> Note: Even without network conditions, the natural state can also
cause packet reordering, especially in public cloud platforms such as
AWS EC2.
> Note: You can use command `curl
http://localhost:2023/tc/api/v1/config/raw -X POST -d 'tcdel --all
ens5'` to reset the network condition settings.
Check the web console, you will see the reordering setup:
<img width="500" alt="TC Settings"
src="https://github.com/user-attachments/assets/b278fdf4-9fcc-4aac-b534-dfa34e28c371"
/>
Then, publish stream via WHIP: http://localhost:8080/players/whip.html
And, play via HTTP-FLV: http://localhost:8080/players/srs_player.html
Finished by AI:
* [AI: Extract audio jitter buffer to class
AudioPacketCache](a4097d9374)
* [AI: Add utest and fix
bug.](c919227af5)
---------
Co-authored-by: Haibo Chen <495810242@qq.com>
Co-authored-by: winlin <winlinvip@gmail.com>
## Introduce
This PR adds support for viewing streams via the RTSP protocol. Note
that it only supports viewing streams, not publishing streams via RTSP.
Currently, only publishing via RTMP is supported, which is then
converted to RTSP. Further work is needed to support publishing RTC/SRT
streams and converting them to RTSP.
## Usage
Build and run SRS with RTSP support:
```
cd srs/trunk && ./configure --rtsp=on && make -j16
./objs/srs -c conf/rtsp.conf
```
Push stream via RTMP by FFmpeg:
```
ffmpeg -re -i doc/source.flv -c copy -f flv rtmp://localhost/live/livestream
```
View the stream via RTSP protocol, try UDP first, then use TCP:
```
ffplay -i rtsp://localhost:8554/live/livestream
```
Or specify the transport protocol with TCP:
```
ffplay -rtsp_transport tcp -i rtsp://localhost:8554/live/livestream
```
## Unit Test
Run utest for RTSP:
```
./configure --utest=on & make utest -j16
./objs/srs_utest
```
## Regression Test
You need to start SRS for regression testing.
```
./objs/srs -c conf/regression-test-for-clion.conf
```
Then run regression tests for RTSP.
```
cd srs/trunk/3rdparty/srs-bench
go test ./srs -mod=vendor -v -count=1 -run=TestRtmpPublish_RtspPlay
```
## Blackbox Test
For blackbox testing, SRS will be started by utest, so there is no need
to start SRS manually.
```
cd srs/trunk/3rdparty/srs-bench
go test ./blackbox -mod=vendor -v -count=1 -run=TestFast_RtmpPublish_RtspPlay_Basic
```
## UDP Transport
As UDP requires port allocation, this PR doesn't support delivering
media stream via UDP transport, so it will fail if you try to use UDP as
transport:
```
ffplay -rtsp_transport udp -i rtsp://localhost:8554/live/livestream
[rtsp @ 0x7fbc99a14880] method SETUP failed: 461 Unsupported Transport
rtsp://localhost:8554/live/livestream: Protocol not supported
[2025-07-05 21:30:52.738][WARN][14916][7d7gf623][35] RTSP: setup failed: code=2057
(RtspTransportNotSupported) : UDP transport not supported, only TCP/interleaved mode is supported
```
There are no plans to support UDP transport for RTSP. In the real world,
UDP is rarely used; the vast majority of RTSP traffic uses TCP.
## Play Before Publish
RTSP supports audio with AAC and OPUS codecs, which is significantly
different from RTMP or WebRTC.
RTSP uses commands to exchange SDP and specify the audio track to play,
unlike WHEP or HTTP-FLV, which use the query string of the URL. RTSP
depends on the player’s behavior, making it very difficult to use and
describe.
Considering the feature that allows playing the stream before publishing
it, it requires generating some default parameters in the SDP. For OPUS,
the sample rate is 48 kHz with 2 channels, while AAC is more complex,
especially regarding the sample rate, which may be 44.1 kHz, 32 kHz, or
48 kHz.
Therefore, for RTSP, we cannot support play-then-publish. Instead, there
must already be a stream when playing it, so that the audio codec is
determined.
## Opus Codec
No Opus codec support for RTSP, because for RTC2RTSP, it always converts
RTC to RTMP frames, then converts them to RTSP packets. Therefore, the
audio codec is always AAC after converting RTC to RTMP.
This means the bridge architecture needs some changes. We need a new
bridge that binds to the target protocol. For example, RTC2RTMP converts
the audio codec, but RTC2RTSP keeps the original audio codec.
Furthermore, the RTC2RTMP bridge should also support bypassing the Opus
codec if we use enhanced-RTMP, which supports the Opus audio codec. I
think it should be configurable to either transcode or bypass the audio
codec. However, this is not relevant to RTSP.
## AI Contributor
Below commits are contributed by AI:
* [AI: Remove support for media transport via
UDP.](755686229f)
* [AI: Add crutial logs for each RTSP
stage.](9c8cbe7bde)
* [AI: Support AAC doec for
RTSP.](7d7cc12bae)
* [AI: Add option --rtsp for
RTSP.](f67414d9ee)
* [AI: Extract SrsRtpVideoBuilder for RTC and
RTSP.](562e76b904)
---------
Co-authored-by: Jacob Su <suzp1984@gmail.com>
Co-authored-by: winlin <winlinvip@gmail.com>
For H.264, only when the NAL Type is 1, 2, 3, or 4 is it possible for
B-frames to be present; that is, non-IDR pictures and slice data.
The current `SrsVideoFrame::parse_avc_bframe()` function uses incorrect
logic to determine if a NALU can contain B-frames. The original
implementation only checked for specific NALU types (IDR, SPS, PPS) to
mark as non-B-frames, but this approach misses many other NALU types
that cannot contain B-frames according to the H.264 specification.
According to H.264 specification (ISO_IEC_14496-10-AVC-2012.pdf, Table
7-1), B-frames can **only** exist in these specific NALU types:
- Type 1: Non-IDR coded slice (`SrsAvcNaluTypeNonIDR`)
- Type 2: Coded slice data partition A (`SrsAvcNaluTypeDataPartitionA`)
- Type 3: Coded slice data partition B (`SrsAvcNaluTypeDataPartitionB`)
- Type 4: Coded slice data partition C (`SrsAvcNaluTypeDataPartitionC`)
All other NALU types (IDR=5, SEI=6, SPS=7, PPS=8, AUD=9, etc.) cannot
contain B-frames by definition.
---------
Co-authored-by: Jacob Su <suzp1984@gmail.com>
Co-authored-by: winlin <winlinvip@gmail.com>
This PR refactors the RTMP to RTC bridge to support multiple video
tracks by implementing lazy initialization of audio and video tracks.
Instead of pre-determining track parameters during bridge construction,
tracks are now initialized dynamically when the first packet of each
type is received, allowing proper codec detection and track
configuration for dual video track scenarios.
Failed to view WHEP with HEVC before publishing RTMP, because the
default codec is AVC and will not be updated until the stream is
published. This enables better handling of streams with multiple video
tracks in RTMP to WebRTC bridging scenarios. Now, you are able to:
1. View WHEP with HEVC: Play with WebRTC:
http://localhost:8080/players/whep.html?schema=http&&codec=hevc
2. Then publish by RTMP: `ffmpeg -stream_loop -1 -re -i doc/source.flv
-c:v libx265 -profile:v main -preset fast -b:v 2000k -maxrate 2000k
-bufsize 2000k -bf 0 -c:a aac -b:a 48k -ar 44100 -ac 2 -f flv
rtmp://localhost/live/livestream`
Or publish RTMP with HEVC, then view by WHEP.
Note that if the codecs do not match, the error log will display RTC:
`Drop for ssrc xxxxxx not found`. For example, this can occur if you
publish RTMP with HEVC while viewing the stream with AVC.
**Introduce**
This pull request builds upon the foundation laid in
https://github.com/ossrs/srs/pull/4289 . While the previous work solely
implemented unidirectional HEVC support from RTMP to RTC, this
submission further enhances it by introducing support for the RTC to
RTMP direction.
**Usage**
Launch SRS with `rtc2rtmp.conf`
```bash
./objs/srs -c conf/rtc2rtmp.conf
```
**Push with WebRTC**
Upgrade browser to Chrome(136+) or Safari(18+), then open [WHIP
encoder](http://localhost:8080/players/whip.html?schema=http&&codec=hevc),
push stream with URL that enables HEVC by query string `codec=hevc`:
```bash
http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream&codec=hevc
```
This query string `codec=hevc` is used to select the video codec, and
generate lines in the answer SDP.
```
m=video 9 UDP/TLS/RTP/SAVPF 49 123
a=rtpmap:49 H265/90000
```
The encoder log also show the codec:
```
Audio: opus, 48000HZ, channels: 2, pt: 111
Video: H265, 90000HZ, pt: 49
```
**Play with RTMP**
Play HEVC stream via RTMP.
```bash
ffplay -i rtmp://localhost/live/livestream
```
You will see the codec in logs:
```
Stream #0:0: Audio: aac (LC), 48000 Hz, stereo, fltp
Stream #0:1: Video: hevc (Main), yuv420p(tv, bt709), 320x240, 30 fps, 30 tbr, 1k tbn
```
You can also use [WHEP
player](http://localhost:8080/players/whep.html?schema=http&&codec=hevc)
to play the stream.
Important refactor with AI:
* [AI: Refactor packet cache for RTC frame
builder.](b8ffa1630e)
* [AI: Refactor the packet copy and free for
SrsRtcFrameBuilder](f3487b45d7)
* [AI: Refactor the frame detector for
SrsRtcFrameBuilder](4ffc1526b9)
* [AI: Refactor the packet_video_rtmp for
SrsRtcFrameBuilder](81f6aef4ed)
* [AI: Add utests for
SrsCodecPayload.codec](61eb1c0bfc)
* [AI: Add utests for VideoPacketCache in
SrsRtcFrameBuilder.](fd25480dfa)
* [AI: Add utests for VideoFrameDetector in
SrsRtcFrameBuilder.](b4aa977bbd)
* [AI: Add regression test for RTC2RTMP with
HEVC.](5259a2aac3)
---------
Co-authored-by: Jacob Su <suzp1984@gmail.com>
Co-authored-by: winlin <winlinvip@gmail.com>
1. It cannot retrieve codec information on `Firefox` by
`getSenders/getReceivers`
2. It can retrieve codec information on `Chrome` by `getReceivers`, but
incorrect, like this:
3. So, we retrieve codec information from `getStats`, and it works well.
4. The timer is used because sometimes the codec cannot be retrieved
when `iceGatheringState` is `complete`.
5. Testing has been completed on the browsers listed below.
- [x] Chrome
- [x] Edge
- [x] Safari
- [x] Firefox
---------
Co-authored-by: winlin <winlinvip@gmail.com>
1. When the chunk message header employs type 1 and type 2, the extended
timestamp denotes the time delta.
2. When the DTS (Decoding Time Stamp) experiences a jump and exceeds
16777215, there can be errors in DTS calculation, and if the audio and
video delta differs, it may result in audio-video synchronization
issues.
---------
`TRANS_BY_GPT4`
---------
Co-authored-by: 彭治湘 <zuolengchan@douyu.tv>
Co-authored-by: Haibo Chen(陈海博) <495810242@qq.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: winlin <winlinvip@gmail.com>
In the scenario of converting WebRTC to RTMP, this conversion will not
proceed until an SenderReport is received; for reference, see:
https://github.com/ossrs/srs/pull/2470.
Thus, if HTTP-FLV streaming is attempted before the SR is received, the
FLV Header will contain only audio, devoid of video content.
This error can be resolved by disabling `guess_has_av` in the
configuration file, since we can guarantee that both audio and video are
present in the test cases.
However, in the original regression tests, the
`TestRtcPublish_HttpFlvPlay` test case contains a bug:
5a404c089b/trunk/3rdparty/srs-bench/srs/rtc_test.go (L2421-L2424)
The test would pass when `hasAudio` is true and `hasVideo` is false,
which is actually incorrect. Therefore, it has been revised so that the
test now only passes if both values are true.
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: winlin <winlinvip@gmail.com>
To enable H.265 support for the WebRTC protocol, upgrade the pion/webrtc
library to version 4.
---------
Co-authored-by: john <hondaxiao@tencent.com>
Co-authored-by: winlin <winlinvip@gmail.com>
