I might take over one of these one-year-free hosted Lemmy instances on my server infrastructure, but I read several times now that Lemmy’s image hosting system Pict-rs is using a lot of storage quickly.
The server I could run this on is limited to 32gb ssd storage with no easy way to expand it.
Is there some way to limit the image storage use and automatically prune old images that are not user or community icons or such?
Storage requirements depend entirely on the amount of images that users upload. In case of slrpnk.net, there are currently 1.6 GB of pictrs data. You can also use s3 storage, or something like sshfs to mount remote storage.
How much of that is cached from federated instances though? I can hardly imagine a low-traffic community like that uploaded 1.6GB of their own images already. If it is mostly cached then that can increase very quickly as new users subscribe to additional communities on other servers.
There is no caching, images from other instances are loaded directly from the remote server by your browser.
I see, well that is one risk less then. I guess with automatic down-scaling in pict-rs 0.4 it will be mostly solved as there will not be a bunch of 5mb direct uploads.
Edit: well thumbnails at least are definitely cached, larger images too, I just tested it on slrpnk.net Edit: odd, but not all of them. Something is strange… Ah I think I know what is happening… actual user uploads do not get cached, but images from linked websites do, even if the origin is a federated instance. But those website images are usually quite well optimized.
Now you know why there needs to be a decentralized picture storage hosting that works for the web, in the same way torrents do for even larger data like video.
You have tons of servers hosting the exact same pictures needlessly while sharing none of the hosting costs.
That was the original idea of IPFS, no? Just that they pivoted now to trying to sell you filecoins :(
I’m not against including an ipfs layer in pict-rs, but the complexity would go way up. Federating an image between lemmy servers would require sending the ipfs uri between servers via activitypub, and then each receiving server sending that uri to pict-rs. pict-rs would then need to decide, on each server, if the ipfs-stored image matches that servers’ image requirements (max filesize, max dimensions, etc), and if it does, then that pict-rs server would request to pin the image. I don’t know exactly how ipfs pinning works, but ideally it would only be stored locally if it isn’t already stored in at least N other locations. If the remote image doesn’t match the local server’s configuration, it could either be rejected or downloaded & processed (resized etc).
Serving ipfs-stored images that aren’t replicated locally might also be slow, but I won’t know for sure unless I actually try building this out.
deleted by creator
This seems to be the case right now, yes.
The great thing is that deduplication would also be built-in with the IPFS layer, apart of the obvious advantages.
Oh, you said it too.
Yeah I think so, but I have no idea how “trust” works in IPFS.
In torrents, you have to explicitly be seeding that torrent: if you don’t want to seed the file(s), you remove the torrent. With IPFS I think people can just throw whatever in there.
Pictrs over IPFS could be a good start.