Improving workflow with proxies

ljm96
ljm96 Posts: 3 Just Starting Out

Hi. I have been running Hitfilm Express on my laptop for over a year now. Lately, I have been working with larger files that almost require me to generate a proxy in order to be able to scrub the timeline with minimal lag.

These files can be as long as 7 hours, and at 1080p, the proxy size can get as big as 20GB. The size of these files isn't so much of an issue when saving to my 4TB external hard drive (internal SSD is only 512GB). However, I am finding that these proxies can take as long as 12 hours each to generate. This is okay when I only need one of these files in my project, but when I need more - 5 or even 10 - this can be problematic.

I did notice that generating these proxies to the external hard drive would occasionally cause problems, and sometimes the proxy would not load in the project even when the proxy most definitely exists.

Is there anything I can do to make these proxies generate faster without errors? I suspect saving to the SSD would provide better results, but is there also anything I can do to my system to speed up the process? Thanks.

Best Answer

  • NormanPCN
    NormanPCN Posts: 4,298 Expert
    Answer ✓

    Sounds like I'm better off using the quality proxy option then (if I decide to keep using proxies that is).

    Certainly, if the quality encode is on your machine as it is on mine. A quality proxy will be a lot bigger than a performance proxy. The quality proxy might be bigger than your source media.

    No activity when running the video during playback.

    Ouch. Then you really do want a Fast decode AVC type of encode (like my AVC suggested settings). With software decode those settings will be better.

    That the GPU graph for video decode is listed seems to say Windows thinks the GPU has a video decoder (it does by my reading of the specs). You could try playing the video in the windows video player and check the video decode usage at that point. It could be Hitfilm does not support your AMD video decoder iteration, or not video decoders in AMD APUs (only discrete GPUs). Hitfilm supposedly does support AMD AVC video decoders.

    The video has been downloaded using software called Twitch Leecher

    Okay, I suspected it might have been a streaming encode from the specs.

    --

    I would still recommend a one time transcode of the source, away from VFR, and work with that. You could dump the original at that point.

Answers

  • NormanPCN
    NormanPCN Posts: 4,298 Expert

    Is there anything I can do to make these proxies generate faster without errors? 

    ...but is there also anything I can do to my system to speed up the process?

    Faster without errors. Speed up the process. Not likely.

    You did not mention the specific version of Express you are running. It can matter for discussion.

    You did not mention about the proxy type you are creating. Quality or Performance. The Performance will be smaller by tons but they can look like ass (IMO, may depend on Hitfilm version).

    Also, external hard drives are kinda slow. Maybe necessary in your case, but understand the compromise.

    The Hitfilm proxy system, is a proxy system, but that is probably about all one can say. Cough, cough. IMO. The Quality proxies are excessive and the Performance ones can look terrible. Neither one is really optimized for hardware decode. The encode setting rules are different for software versus hardware decode. Not big differences in encode settings here but it can help. A proxy after all is about optimized performance.

    Really you can do better making your own proxies. The proxy can be smaller and performance better in decode and look as good or better (esp Performance proxies). But Hitfilm is not good for doing this manual proxy switch. Media does not auto conform itself to the timeline dimensions, or have the ability to enable such a feature. You end up doing a lot of manual clip/layer resizing on a proxy <-> source media change.

    Really, you are also better off splitting those long files into smaller ones. Hitfilm has always seemed to have issues with long files. Splitting those files losslessly can be done. I've tested that with ffmpeg or shareware tools like Lossless cut. Maybe 30 minute pieces.

    Depending on your source media specs, you may be better off just doing a single transcode to a result file that would decode and scrub better than your source media. I would still split the long media. Then you can delete the source media after the transcode and things seem fine.

    Ultimately I think you should be able to be able to get good timeline performance with a simple, but good, single transcode. A two media, proxy and source, setup may not be required.

    If you can post a text view MediaInfo report here then we can know what you are working with as source media. (it's a free utility program).

    Post your machine specs as well so we can have a little idea what we are working with.

  • ljm96
    ljm96 Posts: 3 Just Starting Out
    Thanks for the reply! Please see answers below.

    HitFilm Express version 2021.3

    I am currently using Performance for my proxy type.

    MediaInfo report:
    Format : MPEG-4
    Format profile : Base Media
    Codec ID : isom (isom/iso2/avc1/mp41)
    File size : 15.3 GiB
    Duration : 7 h 1 min
    Overall bit rate : 5 182 kb/s
    Writing application : Lavf59.16.100

    Video
    ID : 1
    Format : AVC
    Format/Info : Advanced Video Codec
    Format profile : [email protected]
    Format settings : CABAC / 4 Ref Frames
    Format settings, CABAC : Yes
    Format settings, Reference frames : 4 frames
    Format settings, GOP : M=4, N=120
    Codec ID : avc1
    Codec ID/Info : Advanced Video Coding
    Duration : 7 h 1 min
    Source duration : 7 h 1 min
    Bit rate : 5 003 kb/s
    Width : 1 920 pixels
    Height : 1 080 pixels
    Display aspect ratio : 16:9
    Frame rate mode : Variable
    Frame rate : 60.000 FPS
    Minimum frame rate : 20.000 FPS
    Maximum frame rate : 62.500 FPS
    Color space : YUV
    Chroma subsampling : 4:2:0
    Bit depth : 8 bits
    Scan type : Progressive
    Bits/(Pixel*Frame) : 0.040
    Stream size : 14.7 GiB (97%)
    Source stream size : 14.7 GiB (97%)
    Color range : Limited
    Color primaries : BT.709
    Transfer characteristics : BT.709
    Matrix coefficients : BT.709
    Codec configuration box : avcC

    Audio
    ID : 2
    Format : AAC LC
    Format/Info : Advanced Audio Codec Low Complexity
    Codec ID : mp4a-40-2
    Duration : 7 h 1 min
    Bit rate mode : Constant
    Bit rate : 161 kb/s
    Channel(s) : 2 channels
    Channel layout : L R
    Sampling rate : 48.0 kHz
    Frame rate : 46.875 FPS (1024 SPF)
    Compression mode : Lossy
    Stream size : 485 MiB (3%)
    Default : Yes
    Alternate group : 1

    OS Name Microsoft Windows 10 Home
    Version 10.0.19044 Build 19044
    Other OS Description Not Available
    OS Manufacturer Microsoft Corporation
    System Name LAPTOP-HSD73VGS
    System Manufacturer Acer
    System Model Aspire E5-523G
    System Type x64-based PC
    System SKU Aspire E5-523G_1099_1.08
    Processor AMD A9-9410 RADEON R5, 5 COMPUTE CORES 2C+3G, 2900 Mhz, 2 Core(s), 2 Logical Processor(s)
    BIOS Version/Date Insyde Corp. V1.08, 11/07/2016
    SMBIOS Version 3.0
    Embedded Controller Version 1.60
    BIOS Mode UEFI
    BaseBoard Manufacturer Acer
    BaseBoard Product Wasp_SR
    BaseBoard Version V1.08
    Platform Role Mobile
    Secure Boot State On
    PCR7 Configuration Elevation Required to View
    Windows Directory C:\WINDOWS
    System Directory C:\WINDOWS\system32
    Boot Device \Device\HarddiskVolume1
    Locale United States
    Hardware Abstraction Layer Version = "10.0.19041.1806"
    Installed Physical Memory (RAM) 16.0 GB
    Total Physical Memory 15.5 GB
    Available Physical Memory 9.90 GB
    Total Virtual Memory 18.5 GB
    Available Virtual Memory 11.3 GB
    Page File Space 3.00 GB
    Page File C:\pagefile.sys
    Kernel DMA Protection Off
    Virtualization-based security Not enabled
    Device Encryption Support Elevation Required to View
    Hyper-V - VM Monitor Mode Extensions Yes
    Hyper-V - Second Level Address Translation Extensions Yes
    Hyper-V - Virtualization Enabled in Firmware No
    Hyper-V - Data Execution Protection Yes
  • NormanPCN
    NormanPCN Posts: 4,298 Expert
    edited September 11

    Executive summary. I think we can get you going without all that Hitfilm proxy BS. Just a simple transcode of your source media and just use that.

    System SKU Aspire E5-523G_1099_1.08

    Processor AMD A9-9410 RADEON R5, 5 COMPUTE CORES 2C+3G, 2900 Mhz, 2 Core(s), 2 Logical Processor(s)

    This is a pretty low end CPU+GPU, but I think we can keep you going. 60fps video (1080p60) will be harder for this but you hopefully can still be okay. Kicking the viewer in half rez can certainly help with performance of edit effect/transitions and all that. If needed.

    HitFilm Express version 2021.3

    I don't have 2021.3 installed and I probably will not for a test, but the current Hitfilm Free 2022.1 is the subsequent version of this so should be similar. These versions (>= 2021.2) of the free Hitfilm are using the Windows decoders for AVC video and AAC audio. There are often a lot of problems here. Reports from forum, and it does not take much for me to generate something Hitfilm chokes on. Problems probably mostly in how Hitfilm is attempting to use the OS subsystem (aka badly). Nothing you can do about this, but hopefully try to work around it when it surfaces.

    These Hitfilm versions also use the OS encoders (AVC and AAC). This is where the proxy generation you are attempting to use comes in. I cannot comment on your AMD APU setup, but in tests on my Nvidia GPU setup....

    You mentioned the time to generate proxies. Quality proxies should generate faster than Performance proxies. Dramatically better on Nvidia systems. The GPU AVC encoder is used. If Hitfilm cannot/does not use the AMD AVC encoder then I cannot comment on Quality proxy setting on your machine. Performance proxies generate hideously slow compared to quality, they are prone to failure, and they can look like ass (poor visual quality). Also, when the first Hitfilm version with proxies came out the performance proxy encode settings were kinda silly, but not the dumb ass stuff I see now (2022.1). In your case they would not timeline scrub a ton better, whereas a quality proxy would (again encode settings used).

    Hitfilm Pro changes the AVC decode and encode setup.

    One thing I wonder if if Hitfilm uses the AMD GPU AVC video decoder on your machine. If you could play your source media and look at the Windows task manager, performance tab, GPU tab, and see if the graph for "video decode" shows activity during playback. You can't switch to the task manager during playback as Hitfilm will stop playback. The graph does show use over time, or you can hover the task manager icon on the taskbar during playback.

    Format settings, GOP : M=4, N=120

    As some might say...well there is your problem. This is a very long GOP (group of pictures)(120 frames, or 2 seconds). It will affect timeline scrubbing performance. Worst case one has to decode 119 frames to display 1 frame. Best case 1 frame. Average 60 frames. Hitfilm proxies do try to run short GOPs. In fact the quality proxy uses the 8 frame GOP I've recommended for years. "NormanAVC" as the term has been coined on the forum.

    Bit rate : 5 003 kb/s

    Width : 1 920 pixels

    Height : 1 080 pixels

    That is a pretty low bitrate for 1080p60. Less than what Youtube allows for 1080p60. Easy to decode for sure but I wonder about the visual quality. Maybe this file came from streaming. Streaming bitrates are often low. Often why people move to 720p60 it keep visual quality up within a limited bitrate. Or just stream 30p. But I digress...

    Frame rate mode : Variable

    Frame rate : 60.000 FPS

    Minimum frame rate : 20.000 FPS

    Maximum frame rate : 62.500 FPS

    Okay, this has got to change. You are using variable frame rate video (VFR). Editors do not like this and the Hitfilm history with VFR can be suspect. Maybe the Hitfilm attempt at handling VFR has been improved. I just do not trust Hitfilm to not be doing seeks during VFR playback. Hitfilm has a history of doing decoder seeks when it should not. Some has been cleaned up. I would hope all, but I would not place a bet. The seeks can kill timeline performance.

    VFR video often comes from screen captures. Even without issues, it is still best to avoid VFR video like the plague. A transcode from the VFR source to something is seriously recommended. In this process one can also generate a file which scrubs well and plays well. Win, Win.

    Writing application : Lavf59.16.100

    What software generated this media file? It seems to be using the ffmpeg core libraries. If the file is direct from the screen capture software, then you may be able to use settings that will edit well without any transcode. Most capture software always generates VFR, but OBS does generate constant frame rate if the output is to an MP4 container. Most using OBS seem to capture to an MKV file and then remux to MP4. Well, the MKV container is basically incapable of encoding CFR video. This is not true of MP4 containers. But if the file is/was about streaming then best encode settings for streaming should take precedence (this bitrates are so low in that circumstance).

    ---

    I will link my (very) old thread on AVC transcoding. That was all about software decoding and the 'rule" can change slightly if a hardware decoder is used, but that last bit is about squeezing all available performance out of the GPU, and your source media is pretty low bitrate so I am not sure it is worth considering.

    Since you are targeting 60p or maybe 59.94p and since Handbrake is the often recommended free tool to transcode, and because I am me, I need to state the following. Handbrake is incapable of generating CFR video for some frame rates. It is just flat out laziness on their part. For playback it never matters, but we are editors here, and this is Hitfilm. 59.94 is one of those Handbrake problem framerates (23.98 is another). 60.0 is okay.

    Also, the CRF 20 value is recommend may be too much for your low bitrate video. You can probably/maybe get away with a larger CRF value. A lower CRF value (higher quality) can increase your file size a lot. Sorry, that the AVC encoder CRF (constant rate factor) term/acronym is so close to CFR (constant frame rate). It can be confusing at first.

    You may still want to split the long file into shorter pieces. I have read long file problem reports, this forum, over time, but I've never been able to personally reproduce (not that I put major effort into do so).

    Again, summary. Ditch the Hitfilm cough proxy system and generate and use something better than what they do. Single file. Included in this is getting away from VFR video. Simple?



  • ljm96
    ljm96 Posts: 3 Just Starting Out
    Thank you for the detailed response. I will make sure to take everything you said into consideration.

    > In your case they would not timeline scrub a ton better, whereas a quality proxy would (again encode settings used).

    Sounds like I'm better off using the quality proxy option then (if I decide to keep using proxies that is).

    > If you could play your source media and look at the Windows task manager, performance tab, GPU tab, and see if the graph for "video decode" shows activity during playback.

    No activity when running the video during playback.

    > That is a pretty low bitrate for 1080p60. Less than what Youtube allows for 1080p60. Easy to decode for sure but I wonder about the visual quality. Maybe this file came from streaming. Streaming bitrates are often low. Often why people move to 720p60 it keep visual quality up within a limited bitrate. Or just stream 30p. But I digress...

    The video has been downloaded using software called Twitch Leecher, which downloads the VOD directly from Twitch. I am then wanting to edit this for YouTube, while maintaining the best possible quality.