As was foretold, we've added advertisements to the forums! If you have questions, or if you encounter any bugs, please visit this thread: https://forums.penny-arcade.com/discussion/240191/forum-advertisement-faq-and-reports-thread/
Options

Choosing an external hard drive/NAS solution

#pipe#pipe Cocky Stride, Musky odoursPope of Chili TownRegistered User regular
Hello H/A!
I am a professional graphic designer and photographer working for a wholesaling/importing company and I need help with file storage.

I work on a mac which is connected by ethernet to an office-wide fileserver. I Currently keep all my working files on my mac, but my boss would like to move them onto the server so that it's accessible by everyone who needs it. It's a fair request.

The only problem is, I'm working with very large working files - these aren't word docs and spreadsheets, they're 200 page InDesign documents and 3'x8' vector Illustrator banners. My average working file is about 100mb and the bigger ones are 1gb or more each. Working on these giant files over ethernet generates uber lag and I don't have time to wait 5 minutes everytime I hit cmd-S

So can you help me work out a storage solution? I'm hoping there's something I can use with a USB or Firewire (it's a 2010 mac) connection which can also connect by ethernet to the network? I've been looking at NAS systems, but they seem to be ethernet only?

Posts

  • Options
    bowenbowen How you doin'? Registered User regular
    Sounds like what you're looking for is version control.

    Work locally, push to the server. There's a few different types like git and subversion, and a few commercial. There does exist ethernet based NAS with USB, buffalo was the one I've used in the past

    http://store.buffalotech.com/store/bufftech/en_US/pd/productID.304362500/quantity.1

    Something like that. On the old version we had you could use USB and the network simultaneously. Not sure if you still can.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    #pipe#pipe Cocky Stride, Musky odours Pope of Chili TownRegistered User regular
    I see that version control was designed for coding and a lot of its tutorials are based around coding projects.

    Would it work well with a large amount of larger sized files?

  • Options
    bowenbowen How you doin'? Registered User regular
    I've known a few people that have used it. I don't see any reason it'd be problematic, you won't be able to use a lot of the fancy features like comparing, but it should work okay. The NAS solution should still work though.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    #pipe#pipe Cocky Stride, Musky odours Pope of Chili TownRegistered User regular
    Actually reading up, it seems like a file backup/migration system with specific options would be a good solution.

    I found a windows only program called Autover which detects updated files in specified folders and copies them to a specified location. Something like that for mac would be simple and effective I think

    basically creating a directory on the server which contains a copy all my necessary folders, and updates only the files which are updated on my computer. It could be set to do this every night after I leave - once a day updates would be fine. I don't really need versioning options because I don't need to retain multiple older versions - just saving the latest version is fine.

  • Options
    bowenbowen How you doin'? Registered User regular
    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    BouwsTBouwsT Wanna come to a super soft birthday party? Registered User regular
    Bookmarking this, I've been interested in this topic for some time.

    Between you and me, Peggy, I smoked this Juul and it did UNTHINKABLE things to my mind and body...
  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    #pipe wrote: »
    Hello H/A!
    I am a professional graphic designer and photographer working for a wholesaling/importing company and I need help with file storage.

    I work on a mac which is connected by ethernet to an office-wide fileserver. I Currently keep all my working files on my mac, but my boss would like to move them onto the server so that it's accessible by everyone who needs it. It's a fair request.

    The only problem is, I'm working with very large working files - these aren't word docs and spreadsheets, they're 200 page InDesign documents and 3'x8' vector Illustrator banners. My average working file is about 100mb and the bigger ones are 1gb or more each. Working on these giant files over ethernet generates uber lag and I don't have time to wait 5 minutes everytime I hit cmd-S

    So can you help me work out a storage solution? I'm hoping there's something I can use with a USB or Firewire (it's a 2010 mac) connection which can also connect by ethernet to the network? I've been looking at NAS systems, but they seem to be ethernet only?

    I'm gonna throw some cold water on this. Your 2010 Mac will have USB 2.0 at best, which means it's 480Mb/s. Firewire is 800Mb/s. Gigabit ethernet is 1Gb/s.

    This means that connecting a drive by USB will not be noticeably faster than gigabit ethernet and will probably be slower*, assuming there isn't something very wrong with the network. Then again, maybe your ethernet in the office isn't gigabit, maybe it's only 100Mb, which would be something to talk to your boss about.

    * - Even though the raw speed of gigabit ethernet is faster than USB 2.0, network protocols add a certain amount of overhead. Real-world results will vary.

    I think your second post here has the right idea:
    #pipe wrote: »
    Actually reading up, it seems like a file backup/migration system with specific options would be a good solution.

    I found a windows only program called Autover which detects updated files in specified folders and copies them to a specified location. Something like that for mac would be simple and effective I think

    basically creating a directory on the server which contains a copy all my necessary folders, and updates only the files which are updated on my computer. It could be set to do this every night after I leave - once a day updates would be fine. I don't really need versioning options because I don't need to retain multiple older versions - just saving the latest version is fine.

    There are lots of ways you can do this. I'd look into rsync-based solutions. This may require somebody to install an rsync service on your file server.

    The benefit of rsync is that it is block-level. Rather than syncing an entire file, it syncs only the parts of a file that have changed. This makes it ideal for large, monolithic files.

    If installing an rsync service on your file server is not an option, then almost any modern NAS device will do it. I love Synology devices, I cannot recommend them highly enough.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    #pipe#pipe Cocky Stride, Musky odours Pope of Chili TownRegistered User regular
    This is real good stuff, guys, thank you!

    I am painfully aware of the limitations of my almost 5 year old mac but I don't think replacing it is on the cards in the short term.

    I've been looking at Chronosync, and contemplating getting a dedicated Art Department NAS on which to have synced copies of all of the relevant folders.

    Thoughts?

  • Options
    bowenbowen How you doin'? Registered User regular
    Also a good option. Also even if gigabit has overhead because of TCP/IP nonsense, you're never going to get the full speed of USB2, maybe half if you're lucky.

    not a doctor, not a lawyer, examples I use may not be fully researched so don't take out of context plz, don't @ me
  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    edited October 2014
    #pipe wrote: »
    This is real good stuff, guys, thank you!

    I am painfully aware of the limitations of my almost 5 year old mac but I don't think replacing it is on the cards in the short term.

    I've been looking at Chronosync, and contemplating getting a dedicated Art Department NAS on which to have synced copies of all of the relevant folders.

    Thoughts?

    If you get a dedicated NAS, you don't need a third-party software package. rsync will work fine out of the box between any decent NAS and OSX. It just requires command-line stuff. Here's an easy tutorial: http://www.rsync.net/resources/howto/mac_rsync.html

    The benefit of third-party software over rsync is just having a GUI. But again keep in mind that some backup software will back up an entire file regardless of the size; rsync only backs up the blocks inside of a file that have changed. For very large files, you want that kind of block-level backup.

    But yeah a dedicated NAS is the route I'd go. For example, a Synology DS412+ supports up to four drives. Put in 4x 2TB drives in RAID-5, that gives you 5.5GB of usable space and protection against one drive failure.

    QNAP is another good brand, but I haven't had as much direct experience with them.

    Feral on
    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    FeralFeral MEMETICHARIZARD interior crocodile alligator ⇔ ǝɹʇɐǝɥʇ ǝᴉʌoɯ ʇǝloɹʌǝɥɔ ɐ ǝʌᴉɹp ᴉRegistered User regular
    I'd also double-check with your boss or the IT guys that the network infrastructure is sound.

    Ideally, you want gigabit network switches, with jumbo frame support, and gigabit-capable wiring (CAT-5e or CAT-6).

    Shitty switches or shitty wiring will bite you in the ass.

    every person who doesn't like an acquired taste always seems to think everyone who likes it is faking it. it should be an official fallacy.

    the "no true scotch man" fallacy.
  • Options
    #pipe#pipe Cocky Stride, Musky odours Pope of Chili TownRegistered User regular
    We don't have a Sys Admin or IT specialists on staff. A dude comes in occasionally if we need him to set up something specific, but nobody here fulltime has the chops to manage a command line system.

    I think the extra time needed to back up larger files, especially when it's set to backup overnight, is worth a GUI which I can run myself.

  • Options
    Jebus314Jebus314 Registered User regular
    #pipe wrote: »
    Actually reading up, it seems like a file backup/migration system with specific options would be a good solution.

    I found a windows only program called Autover which detects updated files in specified folders and copies them to a specified location. Something like that for mac would be simple and effective I think

    basically creating a directory on the server which contains a copy all my necessary folders, and updates only the files which are updated on my computer. It could be set to do this every night after I leave - once a day updates would be fine. I don't really need versioning options because I don't need to retain multiple older versions - just saving the latest version is fine.

    This isn't the real benefit of versioning. The benefit is keeping everyone on the same page. In my lab there is basically me and 1 other guy updating the experimental software that gets used on all of the lab computers (maybe 4 total). Before I started keeping track of versions I can not tell you how many times I would run old versions of a program (generating bad data) because I thought the lab computer had the most recent version based on trying to remember the last time I made an edit and the last time I updated the program on that computer. Or how many times there was confusion between what I was telling someone the program would do and what it was actually doing because they had grabbed an older version. Or how many times the one other guy editing the program would make some edit about the same time I had made an edit and then one of our edits would get erased because the other person would "update" everyone with their version.

    Even if no one besides you will ever do any editing, it's worth it to keep track of version numbers so that you can verify exactly what copy someone else has and whether or not it is the most up to date.

    "The world is a mess, and I just need to rule it" - Dr Horrible
  • Options
    useless4useless4 Registered User regular
    I wouldn't mess with a NAS if you have a server setup already in place.
    Easiest possible way is to go to the App Store and buy Folderwatch.
    Setup a local directory where you work and set Folderwatch to automatically mirror it (with some options you could set if desired) to the other server location.
    Done.

    I do this to manage my video production back up files across two different NAS locations.

Sign In or Register to comment.