Archiving old projects: what to keep?

CleverTaglineCleverTagline ModeratorLas Vegas, NVModerator, Website User, Ambassador, HitFilm Beta Tester Posts: 3,039 Ambassador

My 1TB system hard drive is nearly full, and I'm woefully unprepared for how to deal with it.  I knew I'd hit this point sooner or later, but frankly didn't think I'd hit it this soon. However, after transcoding the screen capture files for part two of the animated lines tutorial, I found myself left with just under 700MB of free space. The export for part one was over 1GB, so I'm already scrambling to figure out what I can delete just to have enough space to export this project. I have no idea how I'll free enough space to do another one.

So far I haven't been able to offload any of the project files for the other tutorials I've made. I have a 500MB external drive that's nearly full, and another (1TB, I think) that I bought ages ago and never hooked up, but I can't find the power supply for it. Unfortunately I’m not in a financial position to just go buy another external drive, but I might just have to suck it up and add it to the credit card (which is already close to its max after our move a couple months back).

Anyway, I'm curious to hear what others do with their old project files, especially anyone else who creates tutorials. Right now, the vast majority of my projects are tutorials, so I've got lots of massive transcoded screen captures. Part of me is thinking that once the tutorial is edited and published, those screen captures aren't worth keeping. It's not like I'm going to reuse them for anything, and I could free up gigabytes of space by deleting them. But I'm also slightly paranoid that some totally unforeseen situation might arise where I might want them. Then again, I've been making HitFilm tutorials for a year and a half, and so far that situation has yet to arise.

«1

Comments

  • Triem23Triem23 Moderator Moderator, Website User, Ambassador, Imerge Beta Tester, HitFilm Beta Tester Posts: 18,286 Ambassador

    Apparently my hard drive budget is waaay higher than yours. My PCs Project drive is a terabyte, but right now I have two 4TB external drives for redundant backups of photo projects, another two 4TB drives as redundant backups of video/tutorial projects, another 6 TB of scattered redundant backups for pre-2012, plus another 4TB I've configured as an external RAID, which might become the "work drive" for photos.

    (Then again the Australia/New Zealand trip generated 870 GB in RAW files, and, when I start generating the 32-bit files from my HDR brackets I suddenly am dealing with 120 MB image files)

    Unfortunately if you don't have the budget for drives, you pretty much have no choice but to delete things and raw screen caps are, arguably, the best place to start. Or, let's say you have multiple iterations of files--for sake of argument  something VFR sourced and a transcode. Kill the original, keep the transcode.

    Fortunately HDDs get cheaper all the time. These days you can get a 2TB external for $70 or 4TB for $100. These should be USB 3 portables powered by the USB port, so no power supply to lose. 

  • Stargazer54Stargazer54 Moderator Moderator, Website User, Ambassador, HitFilm Beta Tester Posts: 2,722 Ambassador

    @CleverTagline Yeah, that's a tough one.   Back in the day we would just keep the original footage and base graphics files along with the EDL.  That way if you had a revision you still had all the original elements available.  Never worried about saving ADO moves and such because those could always be easily re-created.

    Today though, I tend to keep all the incremental edit files but throw away incremental graphics work up to maybe the last two versions.  Then just keep the transcoded footage used in the edit in case there is a revision.  Generally speaking, if you have to fix something 6 months from now you are likely to rebuild an effect anyway - perhaps with an improved technique.

    Also, you can look at alternatives for long term storage on personal cloud solutions such as Google Drive, iCloud or Onedrive.  Not a lot of GB, but a good place to keep something you never use but still want a copy of. 

    Occasionally, I'll comb through my drives and just start chucking old work files on projects I still have the master of.  I know I'll never re-edit those and can get space back that way.

  • tddavistddavis Moderator, Website User Posts: 4,182 Moderator
    edited December 2018

    @CleverTagline I don't create tutorials...heck, I barely create, but I am a digital hoarder   I just throw another external into the mix whenever I get low on space (something like 5ish now.)  I say ish because one is just an old pre sata HD (which I've totally forgotten the interface name for) only like 120G.   I've been trying to back up family photos and videos to the OneDrive and pay for the TB there too.  efficient, oh heck no, but beats a room full of moldy newspapers stacked to the ceiling, I guess. 

    Edit:  IDE, just hit me!

  • GrayMotionGrayMotion Website User Posts: 1,368 Enthusiast
    edited December 2018

    Break out .99 a month and get yourself 50G on iCloud until you can afford to get a couple of 3TB G|Drives. (2TB is something like 9.99 a month.) You already have 5g Free...but that's not much.... but .99 or 9.99  can be found in cans in the ditch :-)

  • WhiteCranePhotoWhiteCranePhoto Website User Posts: 896 Enthusiast

    You're struggling with one TB? Just get some external disks like I did... 40 disks ago. :)

    Yes, I have a LOT of hard disks now... 

    It's par for the course when you generate an average of 1 TB of footage in a weekend shoot 

  • Triem23Triem23 Moderator Moderator, Website User, Ambassador, Imerge Beta Tester, HitFilm Beta Tester Posts: 18,286 Ambassador

    @WhiteCranePhoto yeah, but you shoot on REDs and at 8k ;-)

    @tddavis I have a couple of old IDE drives. For under $30 you can get an IDE/USB-3 caddy. I'm going to do that shortly to back those up. I do have a few things going back to 1996 or so I'd like to access again. I also have an ancient laptop with a Firewire port to retrieve data from four or five old Firewire drives I have. 

  • tddavistddavis Moderator, Website User Posts: 4,182 Moderator

    @Triem23 Funny you mention that.  I recently tumbled on one of those and wish listed it in my Amazon acct.  After buying Christmas for the 10 grandkids this month, I'll probably get a IDE one and a SATA one after the New Year.  They look handy as all get out.

  • CleverTaglineCleverTagline Moderator Las Vegas, NVModerator, Website User, Ambassador, HitFilm Beta Tester Posts: 3,039 Ambassador

    @Triem23 "Apparently my hard drive budget is waaay higher than yours."

    Therein lies the difference: you actually have a hard drive budget. I haven't been able to even dream of buying any gear for a loooong time, even something "basic" like a backup drive.

    "Kill the original, keep the transcode."

    Frankly, I was hoping to hear the opposite advice, considering that it's the massive transcodes that are eating up so much space. And also, just to be clear, these are ALL personal projects, and the vast majority are tutorials. I've only done one client video in the past few years, and I would definitely be more inclined to keep as many of those original files as possible. But with HitFilm tutorials, I don't see the value in keeping the screen caps, original or transcoded. I can't see myself five years down the road thinking, "I really wish I had the original screen captures of that tutorial I did back in the spring of 2017." However, that's why I'm asking here if anyone has (for some reason) found themselves in that very situation, in case there's some value in that stuff that's not evident to me at the moment.

    @WhiteCranePhoto "You're struggling with one TB?"

    Well, I wasn't struggling until I began doing so many tutorials, so this pain is essentially self-inflicted. When it comes down to it, I suppose I've been just acting on blind faith that I could get by with what I've got until such time that I could afford a backup solution.

    I'm probably going to take my power-less drive to Best Buy to see if they can help me find a power supply. Far more doable in my current financial situation than getting a new drive, or even paying for cloud storage.

  • Triem23Triem23 Moderator Moderator, Website User, Ambassador, Imerge Beta Tester, HitFilm Beta Tester Posts: 18,286 Ambassador

    Oops, sorry, I didn't think you're probably doing Cineform transcoding. For tutorials I tend to transcode to "Norman AVC" to keep file sizes down. In that case I kill the original screen cap as soon as I finished the transcoding. I'd rather keep the original file I actually edit with rather than the original capture I'd have to re-transcode to use. 

    Workflow thoughts: I use an MMO mouse with several hotkeys and have several key binds defined I can swap in and out. For tutorials I have keys to start/stop recording. I break my session up into several shorter videos easily that way. Basically, whenever I move to a new section (i.e. On a hypothetical updated interface tutorial, each panel has its own file, at least). Often  if I blow a take on a section, I'll delete that segment and go back to do it again. Maybe I've done something for 10 min and screwed up 9 min in. I'll stop recording, make a note of the error  then pick up on a new file. During transcode I'll refer to my notes and trim out the flub on the transcode. Keeps my segments short for editor performance, keeps file size down for storage, and, doing some "pre-edit" at recording and transcode while it's all fresh reduces edit time a couple of days later when I don't remember exactly how a take went.

    That said, once the updated Hit-U tutorials come out I may delete the first-gen versions entirely. As you said, I won't return to those again. Weddings and events, you never know. I recently re-mastered a wedding from 2013. Client had moved and lost the DVD but wanted to have a viewing party for the 5th anniversary. Having the originals, I remastered for them as progressive mp4 since they have a media computer hooked up to a smart TV. 

  • CleverTaglineCleverTagline Moderator Las Vegas, NVModerator, Website User, Ambassador, HitFilm Beta Tester Posts: 3,039 Ambassador

    @Triem23 "Oops, sorry, I didn't think you're probably doing Cineform transcoding."

    I'm not. I'm transcoding to ProRes 422. I can't recall the last time I reviewed the details of the Norman AVC method, but my gut feeling was that it was specific to AVI.  I'm editing with MOV files on a Mac.

    My capture method up until recently was a lot of back and forth between HitFilm a pre-written script in Evernote, and I'd have to edit out the Evernote bits. Pretty time-consuming. Starting with the first of these animated lines tutorials, though, I decided to take a stab at a method that (I believe) Javert talked about some time ago.  I re-time my VO so that I could follow along in real time and get the demo in a single take, but "single take" doesn't mean a single recording. I still split it up into chunks. I think about where I can cut away to some sample material, and use those as break points in recording. For part one of animated lines, there were about ten different pieces. For part two, there are nine so far. It means I don't have to edit out all the back-and-forth between HitFilm and Evernote, but I eat up more time on the front end with all the retiming of the VO, as well as testing the retiming before recording the screen cap. I'm getting better with the retiming on this second part, and I kinda like it better than all the Evernote trimming I used to do, but it's still a trade off.

    I should also point out that my ProRes transcode files are larger than they've been in the past because of FXhome's new requirement to deliver tutorials at 60fps, which means capturing and transcoding at 60fps as well.

  • NormanPCNNormanPCN Website User Posts: 3,947 Enthusiast

    "I can't recall the last time I reviewed the details of the Norman AVC method, but my gut feeling was that it was specific to AVI. "

    AVC/H.264 does not work in an AVI file. AVI is also Windows specific and Hitfilm is not.

    Norman AVC works anywhere typical AVC is used. MP4 and MOV.

  • Triem23Triem23 Moderator Moderator, Website User, Ambassador, Imerge Beta Tester, HitFilm Beta Tester Posts: 18,286 Ambassador

    "... FXhome's new requirement to deliver tutorials at 60fps..."

    Had I forgotten that, or has it been that long since I've done one (4k required upsampling since my PC is 1080p - the same resolution the VFX shots for $200 mil Marvel and Star Wars movies are mastered.)? That's a very silly requirement for what's basically a mouse pointer moving around. 24 would be fine. Still, I'll make another note since I'm getting ready to pitch a couple at Oli and I want to be compliant.

    The particle dissolve tutorial ended up being screencapped to edited VO, but I had an issue at the time where my flipping screen recorders (all three) kept dropping video frames to black anytime I recorded with VO. Necessity. Not my style and felt alien... But I had to record that tutorial some 10 times. Literally! Tech issues meant that single tutorial took over 100 hours to put together. 

  • CleverTaglineCleverTagline Moderator Las Vegas, NVModerator, Website User, Ambassador, HitFilm Beta Tester Posts: 3,039 Ambassador

    @Triem23 I mentioned my concern about the 4K requirement to Oli, and he said I could go ahead and record at 1080p. The official brief I received later included that lower res as a recording option, as well as being the requirement for delivery, but the frame rate has to be  60fps.

  • WhiteCranePhotoWhiteCranePhoto Website User Posts: 896 Enthusiast

    @CleverTagline yep, that'll eat up disk space... what a bummer that you're in the situation. If you don't expect to need the transcoded footage soon, you could archive the originals knowing that you can re-generate the transcoded versions in the future in case you need them.

    I didn't know about the 60fps requirement... that does seem odd.

     

  • NormanPCNNormanPCN Website User Posts: 3,947 Enthusiast

    60p is quite silly for a screen capture tutorial. 30 seems reasonable and is a nice multiple of common PC monitor frame rates for capture. So much static screen in tutorials.

    4K is the hype. @Triem23 @CleverTagline do you guys even have a 4K monitor to capture with?

    Heck you can just capture and edit and export a 1440/1080 30p video and transcode it to UHD and/or 60p. Anything using x264. You'll get even better compression quality with that.

  • CleverTaglineCleverTagline Moderator Las Vegas, NVModerator, Website User, Ambassador, HitFilm Beta Tester Posts: 3,039 Ambassador

    No 4K here. Technically my monitor won't even do 1080p. Its highest native resolution is 1600x900. I'm using a third party tool to force the resolution to 1080p. It obviously looks a little soft because it's a hack, but it does the job.

  • NormanPCNNormanPCN Website User Posts: 3,947 Enthusiast

    @CleverTagline Is this tool scaling the capture to 1080 and then you are editing that 1080 result?

    If yes, you might be better off capturing native (1600x900) and editing native to preserve pristine quality. Then exporting native rez and then transcoding the output result using a tool that gives control over the interpolation algorithm used for the scaling. Some algorithms may do better on an upscale with high contrast edge graphics (e.g. like the UI) and mograph. Something like ffmpeg, which can be a pain, but all you need is a simple single script command to do the trick.

  • CleverTaglineCleverTagline Moderator Las Vegas, NVModerator, Website User, Ambassador, HitFilm Beta Tester Posts: 3,039 Ambassador
    edited December 2018

    @NormanPCN  No, the tool I use is forcing  the GPU to create a 1080P signal and send that to the monitor. What I capture is true 1080P, but the monitor isn’t technically designed for that resolution, so that’s why I said everything looks a little soft. 

  • PalaconoPalacono Website User, HitFilm Beta Tester Posts: 3,442 Enthusiast
    edited December 2018

    60fps and 4k tutorials to watch on a laptop or mobile phone? Hilarious. :D:D:D

    Budget for a USB 3.0 external 4TB drive. A proper 3.5" one with a power supply (don't lose it) rather than a portable 2.5" you can easily lose/drop and that will be slower and more fragile. They're really not that expensive, and are often weirdly cheaper than buying a bare drive and adding your own drive caddy; as manufacturers appear to subsidise them. For name recognition perhaps? I use WD Elements 4TB drives. Cheap as chips.

  • OliThompsonOliThompson Staff Administrator, Website User Posts: 140 Staff

    The current requirements for all YouTube videos is set at 1080p at 60fps.

    There was a period of time when it was 4k but we reduced it due to very few people using 4k screens and the additional file size. The framerate was set at 60fps after some internal deliberation but I understand everyone’s concerns regarding the actual visible difference this makes, especially on something like a mobile phone. This was done as a means of increasing the overall YouTube channel quality, although perhaps the knock-on cons outweigh the pros.

    We’ll happily review this matter again shortly and see it’s it’s worth scaling back to 1080p at 30fps again. Thanks for your thoughts everyone :)

  • PalaconoPalacono Website User, HitFilm Beta Tester Posts: 3,442 Enthusiast

    @OliThompson if you're using the "upload at 4k and get a higher bitrate 1080p version for free" hack, and if YouTube also similarly increase  the bitrate for 60fps clips; then there is some method in your madness. ;)

    See if you still get the benefits of transcoding a 1080p / 30fps to 4k and/or 60fps, and if you do: people can create at 1080p/30 and just do the final conversion pass before uploading, to fool YouTube into not mangling the videos quite as much.

  • CleverTaglineCleverTagline Moderator Las Vegas, NVModerator, Website User, Ambassador, HitFilm Beta Tester Posts: 3,039 Ambassador

    @NormanPCN Thanks for the clarification about the Norman AVC setup. I took it for a spin with Handbrake, and am definitely impressed with the smaller size for my test file:

    Original capture: 127MB
    ProRes 422 transcode: 10GB
    Norman AVC transcode: 235MB

    I'm too deep into the current project to rework all of my capture files with NAVC, but will definitely consider it moving forward!

  • NormanPCNNormanPCN Website User Posts: 3,947 Enthusiast
    edited December 2018

    @CleverTagline You should be able to get the NAVC file size closer to your capture. I am assuming your capture test media is AVC so we are close enough to an Apples to Apples comparo. Maybe Fuji to a Gala, but apples nonetheless. I would not worry too much about file size if you are pleased with your currently result. I only mention this because of the file storage issues that are the source of this thread. Also, a lower AVC bitrate always edits better than a higher bitrate, and this is CPU speed dependent.

    What constant quality (CRF) value did you try? 20? The transcode size increase seems quite large to the source assuming both are AVC.

    What CRF value to recommend was a real problem for me as there are soo many variables to consider. With a typical high/main profile CABAC source file my settings should result in a file a little bigger than the source. With tutorial captures the transcode can be smaller due to better compression which is due to the better detection and handling of static areas. Again, soo many variables.

  • CleverTaglineCleverTagline Moderator Las Vegas, NVModerator, Website User, Ambassador, HitFilm Beta Tester Posts: 3,039 Ambassador

    @NormanPCNYes, I tried a value of 20 for CRF, based on the screenshot in the thread where you outlined everything. Bitrate-wise, the original reads in Quicktime as 3.17Mbit/sec, while the NAVC file is 5.85Mbit/sec. 

    I also discovered that the NAVC transcode was created at a resolution of 1920x1074. For some bizarre reason, all of the 1080p presets in Handbrake have that vertical resolution, and there's no way to change it. I can reduce the height all I want, but I can't make it larger than 1074. I updated Handbrake right before doing the transcode, so it's not a bug from an old version, and I can't find anyone else mentioning this issue online. This is the Mac version of Handbrake, version 1.1.2 if anyone else wants to compare.

     

  • CleverTaglineCleverTagline Moderator Las Vegas, NVModerator, Website User, Ambassador, HitFilm Beta Tester Posts: 3,039 Ambassador

    Disregard the resolution issue. I read on the Handbrake forum about some auto-cropping feature, and with a couple clicks I was able to disable it. Very annoying.

  • NormanPCNNormanPCN Website User Posts: 3,947 Enthusiast

    Handbrake can be a little quirky with its cropping and resolution. Aspect ratio, anamorphic, modulus et al. Most that stuff only means something when you are transcoding DVDs and such. Anamorph. It can get in the way at times.

    I am still curious about the transcode being so big relative to the original. The quantizers of the capture must have been pretty low. This would cause the increase. This would also mean that for that particular source you can use a lower CRF setting. Still a 6Mbps N-AVC bitrate will edit well on any PC.

    For example with my Nvidia hardware capture it is set for ~40Mbps but a Hitfilm capture with only the viewer having movement results in ~8.4Mbps. The 40 is just a guesstimate of Nvidia for typical/common material. They are just using a certain AVC quantizer range for their ~40 guesstimate. The N-AVC transcode, at 20, results in ~5.5Mbps. Lower bitrate, and probably quantizers, but the SSIM is a crazy high 99.x% because x264 is very very dynamic with its bitrate allocation.

    The point is that you can take a super lower quality/bitrate video and transcode it with a CRF of something crazy like 10, and you will get a massic file. Overkill since one can get a high quality transcode with a higher CRF setting with a lower quality source.

    Some of your softness could be due to a lower capture quality bitrate. Maybe you can find a setting up increase that to test if it gets better. If you are forcing the GPU to 1080 then 1080 should be crisp. The monitor/display is doing a downscale to its native 1600x900 and that might induce some softness. That is independent of what the GPU is rendering.

    If you want to try and optimize your transcode to save storage space you can do visual comparisons (difference blend) but for lower effort you can get Handbrake to have x264 output an SSIM value (structural similarity). It is basically a percentage. 1.0 is perfect. .90 is 90% similar. Simply append a ":ssim=1" to the end of the extra options in Handbrake for such a test. Then after a test encode look at the activity log (tool bar button). Scroll to the bottom and find the SSIM metric. Try different RF values and see what SSIM you get for each with your test file(s). Still you have to get some idea what SSIM is good enough.

    For my GoPro 35Mbps 30p log footage a RF of 20 gives about 97.3% SSIM (first person cam mounted to bike). RF 17 (~twice bitrate) gives ~98.3. I have difference bended the transcode with the original. Any diffs are not visible unless greatly magnified (in exposure). I'm good with 97% and the files are only slightly larger than the source. An Nvidia screen capture of the Hitfilm UI at RF 20 gives 99.x% since there is so little movement frame to frame.

  • CleverTaglineCleverTagline Moderator Las Vegas, NVModerator, Website User, Ambassador, HitFilm Beta Tester Posts: 3,039 Ambassador

    @NormanPCN "If you are forcing the GPU to 1080 then 1080 should be crisp. The monitor/display is doing a downscale to its native 1600x900 and that might induce some softness."

    I believe it's the downscale by the monitor that's the cause of the softness. AFAIK the capture is clean.

    I might do more testing to fine-tune the N-AVC setup, but frankly I'm more than happy with the un-tweaked result. Taking a single file from 10GB to 235MB is way more of a win than I expected.

  • CleverTaglineCleverTagline Moderator Las Vegas, NVModerator, Website User, Ambassador, HitFilm Beta Tester Posts: 3,039 Ambassador

    @NormanPCN I shot some footage on my phone to add to this latest project, and ran it all through the N-AVC transcode. I was quite surprised by the results:

    Clip 1: 1080p, 30fps
    Original: 32.3 MB
    N-AVC: 24.9 MB (77% of original)

    Clip 2: 4K, 30fps
    Original: 30 MB
    N-AVC: 197.7 MB (659% of original)

    Clip 3: 1080p, 30fps
    Original: 26.4 MB
    N-AVC: 55.2 MB (209% of original)

    Obviously the content of each video makes a difference in how efficient the transcode will be, but I didn't expect such a wide spread of percentages.

  • Triem23Triem23 Moderator Moderator, Website User, Ambassador, Imerge Beta Tester, HitFilm Beta Tester Posts: 18,286 Ambassador

    @CleverTagline this is, of course, because @NormanPCN is using Quality Settings not a set bitrate. There should be a "compromise" where some of the flags Norman used can be set with a target bitrate instead of quality setting. 

    Incidentally, Justin, you should be able to set up a "Norman AVC profile" for MPEG Streamclip. I know you use that as your primary transcoder, so that would free up the drive space Handbrake is using. Also, since Handbrake screws up 23.97 and 59.97, MPEG Streamclip is a better transcoder, overall. 

  • CleverTaglineCleverTagline Moderator Las Vegas, NVModerator, Website User, Ambassador, HitFilm Beta Tester Posts: 3,039 Ambassador

    @Triem23 "you should be able to set up a "Norman AVC profile" for MPEG Streamclip."

    That's the first thing I tried, but MPEG Streamclip doesn't offer the amount of control over the export settings that Handbrake does, at least not that I could see. Nowhere to add custom flags, no profile setting, and a couple other missing items. If someone knows how/where to tweak MPEG Streamclip to match what @NormanPCN does in Handbrake, I'd like to know.  Frankly, though, I don't mind switching. I rarely work with footage at 23.97 or 59.97, so Handbrake should work just fine for my purposes.

This discussion has been closed.