How to improve scene total FPS

[naserhharbi]

Is there a way to improve the total FPS of the whole scene? it runs around 300FPS when the streaming is disabled but once I enable the streaming then the FPS drops to below 30FPS.

I noticed these 4 instructions are the heaviest because they are done on the CPU

_tempTex.ReadPixels(new Rect(0, 0, Width, Height), 0, 0, false);
_tempTex.Apply();
_grabber.SetTexture2D (_tempTex);
_grabber.Update ();

I tried to run these in a separate thread but does not work. I was thinking maybe rely on the shader to create the texture and send it to gstreamer but I think that is not feasible then thought about having the Camera Target Texture to be on a shared texture using GPU and have a gstreamer pipeline (using CMD terminal) reads that shared texture and send it as a UDP stream.

I am running a scene with multiple streaming cameras and the issue is the FPS drops to 4 FPS so I need to find a way to improve the FPS.

Any thoughts? How much FPS are you getting when you run the scene?

Tested on:

• Intel i9 latest Gen
• Nvidia RTX A2000 12GB
• 64GB RAM
• Windows 11 Pro

Thank you very much.

[naserhharbi]

So, I managed to do so many workarounds to get the scene FPS to an acceptable rate. What I did is the following:

• I do not use the Unity Gstreamer DLL.
• In "OnRenderImage()", I call "AsyncGPUReadback.Request()" to make the method Async. I have it calling "OnCompleteReadback" on completion
• In "OnCompleteReadback()", I get the render texture raw bytes using this "byte[] rawData = request.GetData().ToArray();"
• After that, I used 2 DLLs from a nuget package to convert Raw to JPG (TurboJpegWrapper).
• I added 'turbojpeg.dll' and 'TurboJpegWrapper.dll' to my Unity project.
• I write the JPG byte array onto a local file.
• I use gstreamer (separate CMD window) to stream multiple files to UDP like so: 'gst-launch-1.0 multifilesrc location=frame_%d.jpg loop=true caps="image/jpeg,framerate=25/1" ! jpegdec ! x264enc tune=zerolatency ! rtph264pay ! udpsink host=127.0.0.1 port=7000'
• I only have one jpg file so it loops on the same file.
• Unity keeps overwriting that same jpg file with an updated render texture from the camera.
• to increase performance, I made the Encoding and Writing to File in a separate thread since they do not use any Unity objects.

Now I can stream 4 cameras and the total FPS is averaging around 100FPS.

Basically I convert the Render Texture to JPEG File then stream the file.

[naserhharbi]

void OnRenderImage(RenderTexture source, RenderTexture destination)
{
    // Check if enough time has passed since the last capture.
    timeSinceLastCapture += Time.deltaTime;
    if (timeSinceLastCapture < captureInterval)
    {
        // Skip this frame.
        return;
    }
    // Reset the timer.
    timeSinceLastCapture = 0f;
    // Copy the source RenderTexture to the destination RenderTexture to apply render.
    Graphics.Blit(source, destination);
    // Asynchronously read the pixels from the RenderTexture.
    AsyncGPUReadback.Request(renderTexture, 0, TextureFormat.RGB24, OnCompleteReadback);
}

[naserhharbi]

void OnCompleteReadback(AsyncGPUReadbackRequest request)
{
    if (request.hasError)
    {
        return;
    }

    // Get Raw bytes.
    byte[] rawData = request.GetData<byte>().ToArray();

    // Format the frame name to be in this format: frame_0.jpg.
    string filename = $@"c:\StreamFrames{CameraIndex}\frame_0.jpg";

    // Save the JPG file to disk with a unique name based on the frame index
    Task.Factory.StartNew(() =>
    {
        ConvertThenWriteFrameToFile(rawData, filename);
    });
}

[naserhharbi]

void ConvertThenWriteFrameToFile(byte[] rawData, string filename)
{
    // Encode the raw texture data to JPEG format
    byte[] jpegBytes = EncodeRawToJPG(rawData);
    // Write the JPEG bytes to a file
    WriteAllBytes(filename, jpegBytes);
}

[naserhharbi]

// using TurboJpegWrapper;
byte[] EncodeRawToJPG(byte[] rawData)
{
    byte[] jpegBytes;
    int quality = 50;
    // Encode Raw to JPEG
    using (var jpegCompressor = new TJCompressor())
    {
        jpegBytes = jpegCompressor.Compress(
            rawData,
            0,
            1920,
            1080, TJPixelFormats.TJPF_RGB,
            TJSubsamplingOptions.TJSAMP_420,
            quality,
            TJFlags.NONE
        );
    }
    return jpegBytes;
}

[naserhharbi]

void WriteAllBytes(string fileName, byte[] data)
{
    // Write the byte array to a file
    File.WriteAllBytes(fileName, data);
}

[naserhharbi]

I hope this is helpful and looking forward to further discussion and improvements on this approach. Thank you for reading.

[naserhharbi]

https://www.nuget.org/packages/TurboJpegWrapper/1.5.2?_src=template#versions-body-tab

[naserhharbi]

[naserhharbi]