• sobchak@programming.dev
    link
    fedilink
    English
    arrow-up
    1
    ·
    11 hours ago

    HDDs have horrible random access times, so if you need to process or just copy a lot of small files, say photos, there’s a significant penalty.

      • sobchak@programming.dev
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        11 hours ago

        Rsync, syncthing, backups, mp3s, photos, json files; idk, a lot of tasks involve large amounts of small files. I personally ran into this problem training models on millions of photos. My GPUs would only get up to 25% utilization with mirrored HDDs, so I had to switch to SSDs.

        Edit: the difference is also significant when compiling large projects or just using git. I imagine some game servers need a lot of random accesses too.

        • Gladaed@feddit.org
          link
          fedilink
          English
          arrow-up
          1
          ·
          edit-2
          10 hours ago

          Why are you doing that on a network storage as opposed to on device?

          Also who got millions of photos at home?

          • sobchak@programming.dev
            link
            fedilink
            English
            arrow-up
            1
            ·
            9 hours ago

            Not enough room in the GPU machine for all the HDDs I needed.

            Also who got millions of photos at home?

            People working on biological datasets.

            • Gladaed@feddit.org
              link
              fedilink
              English
              arrow-up
              1
              ·
              edit-2
              7 hours ago

              Why are you doing that recreationally? How are you different from a researcher?

              Why are you using a home server for that?