• sobchak@programming.dev
    link
    fedilink
    English
    arrow-up
    1
    ·
    edit-2
    11 hours ago

    Rsync, syncthing, backups, mp3s, photos, json files; idk, a lot of tasks involve large amounts of small files. I personally ran into this problem training models on millions of photos. My GPUs would only get up to 25% utilization with mirrored HDDs, so I had to switch to SSDs.

    Edit: the difference is also significant when compiling large projects or just using git. I imagine some game servers need a lot of random accesses too.

    • Gladaed@feddit.org
      link
      fedilink
      English
      arrow-up
      1
      ·
      edit-2
      10 hours ago

      Why are you doing that on a network storage as opposed to on device?

      Also who got millions of photos at home?

      • sobchak@programming.dev
        link
        fedilink
        English
        arrow-up
        1
        ·
        9 hours ago

        Not enough room in the GPU machine for all the HDDs I needed.

        Also who got millions of photos at home?

        People working on biological datasets.

        • Gladaed@feddit.org
          link
          fedilink
          English
          arrow-up
          1
          ·
          edit-2
          7 hours ago

          Why are you doing that recreationally? How are you different from a researcher?

          Why are you using a home server for that?