If you think this post would be better suited in a different community, please let me know.
Topics could include (this list is not intending to be exhaustive — if you think something is relevant, then please don’t hesitate to share it):
- Moderation
- Handling of illegal content
- Server structure (system requirements, configs, layouts, etc.)
- Community transparency/communication
- Server maintenance (updates, scaling, etc.)
Cross-posts
We require applications, and most applications we get are extremely low effort and we don’t approve them. If you have open registrations you’ll be doing a lot of moderation for spam.
Run the software that scans images for CSAM. It’s not perfect but it’s something. If your instance freely hosts whatever without any oversight, word will spread and all of a sudden you’re hosting all sorts of bad stuff. It’s not technically illegal if you don’t know about it, but I personally don’t want anything to do with that.
I will add that if you have open registrations you will be a target for spam and trolls, and if you don’t take quick action then some other instances are likely to defederate from your instance.
This depends on the instance, some will have a low tolerance and defederate pretty quickly, some instances will defederate temporarily until the spammers or trolls move to a different instance, and some won’t care. But you likely won’t know it’s happened unless you notice you aren’t getting content from that instance anymore.
One other thing is that if you’re going to run an instance and aren’t already on Matrix, make an account. It’s how instance admins tend to keep in contact with each other.
[…] if you’re going to run an instance and aren’t already on Matrix, make an account. It’s how instance admins tend to keep in contact with each other.
This is good advice.
Run the software that scans images for CSAM.
Which software is that?
It’s called Lemmy-Safety of Fedi-Safety depending on where you look.
One thing to note, I wasn’t able to get it running on a VPS because it requires some sort of GPU.
One thing to note, I wasn’t able to get it running on a VPS because it requires some sort of GPU.
This is good to know. I know that you can get a VPS with a GPU, but they’re usually rather pricey. I wonder if there’s one where the GPU’s are shared, and you only get billed by how much the GPU is used. So if there is an image upload, the GPU would kick on to check it, you get billed for that GPU time, then it turns off and waits for the next image upload.
I don’t think there are services like that, since usually this means deploying and destructing an instance, which takes a few minutes (if you just turn off the instance you still get billed).
Probably the best option would be to have a snapshot, which costs way less than the actual instance, and create from it each day or so yo run on the images since it was last destroyed.This is kind of what I do with my media collection, I process it on my main machine with a GPU, and then just serve it from a low-power one with Jellyfin.
create from it each day or so yo run on the images since it was last destroyed.
Unfortunately, for this usecase, the GPU needs to be accessible in real time; there is a 10 second window when an image is posted for it to be processed [1].
References
- “I just developed and deployed the first real-time protection for lemmy against CSAM!”. @db0@lemmy.dbzer0.com. !div0@lemmy.dbzer0.com. Divisions by zero. Published: 2023-09-20T08:38:09Z. Accessed: 2024-11-12T01:28Z. https://lemmy.dbzer0.com/post/4500908.
- §“For lemmy admins:”
[…]
- fedi-safety must run on a system with GPU. The reason for this is that lemmy provides just a 10-seconds grace period for each upload before it times out the upload regardless of the results. [1]
[…]
- §“For lemmy admins:”
You can actually run it in async model without pictrs safety and just have it scan your newly uploaded images directly from storage. It just doesn’t prevent upload this way, just deletes them.
You’re referring to using only fedi-safety instead of pictrs-safety, as was mentioned in §“For other fediverse software admins”, here, right?
- “I just developed and deployed the first real-time protection for lemmy against CSAM!”. @db0@lemmy.dbzer0.com. !div0@lemmy.dbzer0.com. Divisions by zero. Published: 2023-09-20T08:38:09Z. Accessed: 2024-11-12T01:28Z. https://lemmy.dbzer0.com/post/4500908.
Probably the best option would be to have a snapshot
Could you point me towards some documentation so that I can look into exactly what you mean by this? I’m not sure I understand the exact procedure that you are describing.
The software is setup in such a way that you can run it on your pc if you have a local gpu. It only needs like 2 gb vram
That is a cool feature, but that would mean that all of the web traffic would get returned to my local network (assuming that the server is set up on a remote VPS), which I really don’t want to have happen. There is also the added downtime potential cause by the added point of failure of the GPU being hosted in a much more volatile environment (ie not, for example, a tier 3 data center).
Not all web traffic, just the images to check. With any decent bandwidth, it shouldn’t be an issue for most. It also setup in such a way as to not cause a downtime if the checker goes down.
With any decent bandwidth, it shouldn’t be an issue for most.
It’s not only the bandwidth; I just fundamentally don’t relish the idea of public traffic being directed to my local network.
Not all web traffic, just the images to check.
Ah, yeah, my bad this was a lack of clarity on my part; I meant all image traffic.
It also setup in such a way as to not cause a downtime if the checker goes down.
Oh? Would the fallback be that it simply doesn’t do a check? Or perhaps it could disable image uploads if the checker is down? Something else? Presumably, this would be configurable.
https://github.com/db0/fedi-safety and the companion app https://github.com/db0/pictrs-safety which can be installed as part of your lemmy deployment in the docker-compose (or with a var in your ansible)
If your instance freely hosts whatever without any oversight, word will spread and all of a sudden you’re hosting all sorts of bad stuff. It’s not technically illegal if you don’t know about it, but I personally don’t want anything to do with that.
Yeah, this is my primary concern. I’m hoping that there are established best practices for handling the majority of this sort of unwanted content.
If you have open registrations you’ll be doing a lot of moderation for spam.
Perhaps Captchas are sufficient?
I just checked and we have that turned on, too.
We don’t get a lot of applications. A couple per week, maybe.
The spam is not from bots, it’s people being paid to spam. Captchas absolutely need to be turned on or else you get bots as well, but they don’t stop the spam.
The spam is not from bots, it’s people being paid to spam.
Do you know any specific/official organizations that do this, and/or examples where it’s occured on Lemmy?
Its pretty random outside the Russian misinformation sites (which I haven’t seen in a while, but they probably got better at hiding).
Its hard to give you a link because mods or admins remove the posts or ban the accounts pretty quick most of the time. But there is a new spam account at least every day (I can think of at least two today. Edit: 4). They come in waves so sometimes there are a whole bunch.
That’s probably another thing you need to know. I’m on Lemmy.nz, you’re on sh.it.works. If some new spam account signs up on Lemmy.world and posts to lemm.ee, then if it’s removed by an admin on your instance it is only removed for people on your instance. Everyone else still sees it as your instance is not hosting either the community or the user so it can’t federate our anything to deal with it. The lemm.ee instance could remove the post or comment with the spam in a way that federates out to other instances, but can’t ban the user except for on their instance. Only the Lemmy.world instance can ban the user in a way that federates out to other instances. This is something you’ll get a better understanding of over time.
Lemmy.world has a lot if help so they don’t have issues, but often the spam will come from obscure instances while the admin is asleep and there is no backup, so every other instance has to remove the spam for their own instance. Then you have to work out how to mitigate that for your own instance when you are asleep. Most admins are pretty understanding that this is a hobby and don’t expect everyone to be immediately available, but if you have open registrations then you are likely to be targeted more and need a better plan.
If some new spam account signs up on Lemmy.world and posts to lemm.ee, then if it’s removed by an admin on your instance it is only removed for people on your instance. Everyone else still sees it as your instance is not hosting either the community or the user so it can’t federate our anything to deal with it. The lemm.ee instance could remove the post or comment with the spam in a way that federates out to other instances, but can’t ban the user except for on their instance. Only the Lemmy.world instance can ban the user in a way that federates out to other instances.
This make me think that we should maintain a community curated blocklist in, for example, a Git repository. It could be a list of usernames, and/or a list of instances that are known to be spam that gets updated as new accounts and instances are discovered. Then any instance owner can simply pull the most current version of the blocklist (this could even be done automatically). Once the originating instance blocks the malicious account, they can be removed from the list. This also gives those who have been blocked a centralized method to appeal the block (eg open an issue to create an appeal).
I would honestly have expected something like this to already exist. I think it’s partly the purpose of Fediseer, but I’m not completely sure.
This make me think that we should maintain a community curated blocklist in, for example, a Git repository.
There would be a few problems I can think of with this approach. The first one is who controls it? Whoever that is, you haven’t solved the issue because now instead of only the instance with the user being able to federate the ban now only the maintainer of the git repo can update the ban list.
If you have many people able to update the repo, then the issue becomes a question of how do you trust all these people to never, ever, ever get it wrong? If you ban a user and opt to remove all their content (which you should, with spam), then if you are automating this you end up with the issue of if anyone screws up then how do you get someone’s account unbanned on all those instances? How do you get all their content restored, which is a separate thing and Lemmy currently provides no good way to do this. How do you ensure there are no malicious people with control of the repo but also have enough instances involved to make it worthwhile?
There is a chat room where instance admins share details of spam accounts, and it’s about the best we have for Lemmy at the moment (it works quite well, really, because everyone can be instantly notified but also make their own decisions about who to ban or if something is spam or allowed on their instance - because it’s pretty common that things are not black and white).
I would honestly have expected something like this to already exist. I think it’s partly the purpose of Fediseer, but I’m not completely sure.
Fediseer has a similar purpose but it’s a little different. So far we have been talking about spam accounts set up on various instances, and the time it takes for those mods and admins to remove the spam. But what happens if instead of someone setting up a spam account on an existing instance, they instead create their own instance purely for spamming other instances?
Fediseer provides a web of trust. An instance receives a guarantee from another instance. That instance then guarantees another instance. It creates a web of trust starting from some known good instances. Then if you wish you can choose to have your lemmy instance only federate with instances that have been guaranteed by another instance. Spam instances can’t guarantee each other, because they need an instance that is already part of the web to guarantee them, and instances won’t do that because they risk their own place in the web if they falsely guarantee another instances (say, if one instance keeps guaranteeing new instances that turn out to be spam, they will quickly lose their own guarantee).
Fediseer actually goes further than this, allowing instances to endorse or censure other instances and you can set up your instance to only federate with instances that haven’t been censured or defederate from instances that others have censured for specific reasons (e.g. “hate speech”, “racism”, etc).
It’s quite a cool tool but doesn’t help the original discussion issue of spam accounts being set up on legitimate instances.
The first one is who controls it?
Ideally, nobody. Anyone could make their own blocklist, and one could choose to pull from any of them.
Fediseer provides a web of trust. An instance receives a guarantee from another instance. That instance then guarantees another instance. It creates a web of trust starting from some known good instances. Then if you wish you can choose to have your lemmy instance only federate with instances that have been guaranteed by another instance. Spam instances can’t guarantee each other, because they need an instance that is already part of the web to guarantee them, and instances won’t do that because they risk their own place in the web if they falsely guarantee another instances (say, if one instance keeps guaranteeing new instances that turn out to be spam, they will quickly lose their own guarantee).
How would one get a new instance approved by Fediseer?
If you ban a user […], then if you are automating this you end up with the issue of if anyone screws up then how do you get someone’s account unbanned on all those instances?
The idea would be that if they are automatically banned, then the removal of the user from the list would then cause them to be automatically unbanned. That being said, you did also state:
If you ban a user and opt to remove all their content (which you should, with spam)
How do you get all their content restored
To which I say that I hadn’t considered that the content would be deleted 😜. I was assuming that the user would only be blocked, but their content would still be physically on the server — it would just be effectively invisible.
how do you trust all these people to never, ever, ever get it wrong?
The naively simple idea was that the banned user could open an appeal to get their name removed from the blocklist. Also, keep in mind that the community’s trust in the blocklist is predicated on the blocklist being accurate.
There is a chat room where instance admins share details of spam accounts, and it’s about the best we have for Lemmy at the moment (it works quite well, really, because everyone can be instantly notified but also make their own decisions about who to ban or if something is spam or allowed on their instance - because it’s pretty common that things are not black and white).
Yeah I think I’m more on the side of this, now. The chat is a decent, and workable solution. It’s definitely a lot more hands-on/manual, but I think it’s a solid middle ground solution, for the time being.
We require applications
Is this functionality built into the Lemmy software?
Addendum (2024-11-11T00:32Z):
Ah, yeah, it looks like it is configurable in the admin panel [1].
References
- Lemmy Documentation. join-lemmy.org. Accessed: 2024-11-11T00:35Z. https://join-lemmy.org/docs/users/01-getting-started.html#registration.
- “2. Getting Started”. §“Registration”.
Question/Answer: Instance admins can set an arbitrary question which needs to be answered in order to create an account. This is often used to prevent spam bots from signing up. After submitting the form, you will need to wait for some time until the answer is approved manually before you can login.
- “2. Getting Started”. §“Registration”.
Yeah, it’s just something like “Tell us why you want to join this instance”. If the answer is “to promote my content” or “qq”, for example, they don’t get approved.
It’s done by the Lemmy software.
deleted by creator
- Lemmy Documentation. join-lemmy.org. Accessed: 2024-11-11T00:35Z. https://join-lemmy.org/docs/users/01-getting-started.html#registration.
I would just turn off media uploads entirely. It’s not worth the risk or disk space.
I would just turn off media uploads entirely.
Do you mean also disabling thumbnails? IIUC, pict-rs handles all thumbnail generation [1]. The reason I point this out is that simply disabling image uploads won’t itself stop the generation of thumbnails [2]. There’s also the question of storing/caching images that come from federated servers.
Referencs
- Lemmy Documentation. Accessed: 2024-11-11T01:59Z. https://join-lemmy.org/docs/administration/administration.html.
- “9. Administration”. §“Lemmy Components”. §“Pict-rs”.
Pict-rs is a service which does image processing. It handles user-uploaded images as well as downloading thumbnails for external images.
- “9. Administration”. §“Lemmy Components”. §“Pict-rs”.
- “I just developed and deployed the first real-time protection for lemmy against CSAM!”. @db0@lemmy.dbzer0.com. Published: 2023-09-20T01:38:09-07:00. Accessed: 2024-11-11T02:16Z. https://lemmy.dbzer0.com/post/4500908.
- ¶1
[…] if the content is a link to an external site, lemmy sill caches the thumbnail and stores it in the local pict-rs […].
- ¶1
- Lemmy Documentation. Accessed: 2024-11-11T01:59Z. https://join-lemmy.org/docs/administration/administration.html.
How much server hosting experience do you have? I asked about database preferences over in Self-Hosting once and they basically all said “don’t choose a database ever. Run. Save yourself while there is still time!”
So maybe use a hosting service I guess. Makes you a more difficult target for attacks but also involves your information getting out into the world in direct connection to your instance.
I asked about database preferences over in Self-Hosting once and they basically all said "don’t choose a database ever.
I’m not sure I follow what you mean; Lemmy uses PostgreSQL.
[Using a hosting service] makes you a more difficult target for attacks but also involves your information getting out into the world in direct connection to your instance.
I’m not sure I understand how one’s data would be leaked by the hoster.
Same way things get leaked by Equifax, Twitch, US Bank, etc. You’re most responsible with your information by not having unnecessary accounts or transactions.
Also, most hosts have WhoIs and ICANN registrations for Domains, but you still need a domain regardless. And further than that they might allow subpeonas from various companies who request the info.
they might allow subpeonas from various companies who request the info.
“Allow” is an interesting choice of words. A subpoena is legally binding (depending on the jurisdiction). One could circumvent this by purchasing a domain anonymously, but I’m not currently aware of a reputable domain provider that allows anonymous purchasing of domains.
—
Addendum (2024-11-11T23:38Z):
I just found Njalla which seems to allow anonymous purchasing of domains, but idk how reputable they are.
It comes down to the individual company on whether or not to fight requests for user information. A lot of precedent exists for not complying.
A lot of precedent exists for not complying.
Would you mind citing a case? I’m curious.
NY Times vs Njalla
Njalla does comply with some requests, and was forced to shut down some pirate bay instances at one point, though. Ghost is another privacy domain seller.
Theres also a term for companies called “Bulletproof Registrars.” For example, some Malaysian Registrars apparently don’t have an address and cannot actually recieve most subpoenas.
Mostly VPNs, I don’t know too much about similar cases with server hosts or domain sellers.
NY Times vs Njalla
Do you have an official record of them not complying with an official court-ordered subpoena? I looked into “NYT vs Njalla”, and it seems like it was the NYT making a private request to Njalla under threats of legal action, but no legal action followed [1][2].
References
- “About those threats”. Blog. Njalla. Published: 2018-01-25. Accessed: 2024-11-12T00:33Z. https://njal.la/blog/about-those-threats/.
- “Njalla gives New York Times The Pirate Bay treatment”. Staff Writer. Mybroadband. Published: 2018-01-26. Accessed: 2024-11-12T00:36Z. https://mybroadband.co.za/news/internet/246265-njalla-gives-new-york-times-the-pirate-bay-treatment.html.
- ¶10
TorrentFreak reported that Njalla did not hear back from the New York Times after sending the response.
- ¶10
It comes down to the individual company on whether or not to fight requests for user information.
Wouldn’t this simply be obstruction of justice?
Not every court order is a criminal case.
Sure, but (in the USA) an investigation precedes a criminal case [2], and a court order is part of that. I directly cite, for example, 18 U.S. Code § 1509 - Obstruction of court orders [1]:
Whoever, by threats or force, willfully prevents, obstructs, impedes, or interferes with, or willfully attempts to prevent, obstruct, impede, or interfere with, the due exercise of rights or the performance of duties under any order, judgment, or decree of a court of the United States, shall be fined under this title or imprisoned not more than one year, or both.
References
- “18 U.S. Code § 1509 - Obstruction of court orders”. Legal Information Institute. Cornell Law School. Accessed: 2024-11-12T00:42Z. https://www.law.cornell.edu/uscode/text/18/1509.
- “A Brief Description of the Federal Criminal Justice Process”. FBI. Accessed: 2024-11-12T00:46Z. https://www.fbi.gov/how-we-can-help-you/victim-services/a-brief-description-of-the-federal-criminal-justice-process.
- §“I. The Pretrial Stage”. §“Investigations, Grand Juries, and Arrests”. ¶1.
If a crime is brought to the attention of federal authorities, whether by a victim of the crime or a witness to it (e.g., a bank robbery), a federal law enforcement agency will undertake an investigation to determine whether a federal offense was committed and, if so, who committed it. […]
- §“I. The Pretrial Stage”. §“Investigations, Grand Juries, and Arrests”. ¶1.
Same way things get leaked by Equifax, Twitch, US Bank, etc. You’re most responsible with your information by not having unnecessary accounts or transactions.
This would be low down on my concern for threat levels. At any rate, the only way to get around this would be to either host it on one’s own hardware on one’s own network, or to somehow anonymously purchase a VPS (I am currently unaware of a trustworthy VPS that allows anonymous hosting. I have heard of BitLaunch, but I don’t know how trustworthy it is — do they have the ability to intercept control of the DO Droplet?).
—
Addendum (2024-11-11T23:40Z):
I just found Njalla which seems to allow anonymous purchasing of VPSs, but idk how reputable they are.
Also, most hosts have WhoIs and ICANN registrations for Domains, but you still need a domain regardless.
I’m not sure exactly what you are referring to. I don’t exactly follow how the VPS provider would have any privileged insight into one’s domain registration.
I’m saying if you payed for a service to host your instance remotely. The domain, the site pages, the the database, everything. Then, everything on the domain would be tied to your person and the service providers have a certain power over your instance aside from just turning off your domain. There are more options to not list or to delist from the WhoIs registry for simple domain purchases.
I just have trust issues, you don’t need to mind my crazy ramblings.
I just have trust issues, you don’t need to mind my crazy ramblings.
Concerns about privacy and anonymity are perfectly valid. Ideally, I would want my involvement in a venture like this to be completely anonymous, but there are practical limitations (generally limited by how much added complexity/added risk one wants to put up with).
I’m saying if you payed for a service to host your instance remotely. The domain, the site pages, the the database, everything. Then, everything on the domain would be tied to your person and the service providers have a certain power over your instance aside from just turning off your domain.
Ah, okay, I was under the assumption that the domain was purchased through a separate, independent provider, rather than through the same provider as that of the VPS.
How much server hosting experience do you have?
I’ve never hosted a public facing social media service. I have a few years experience hosting a number of my own personal services, but they aren’t at the scale of a public facing Lemmy instance.
You should be good as long as you know PostgreSQL
Aha, well, it depends on what you mean by “know”.
It could be costly in a few places, so choose your host wisely:
- data ingress/egress
- storage (block and DB)
- load balancing (if you choose to go that route)
I know that R2 has no charge for ingress/egress.
The block and db costs are technically unbounded, and will never decrease by default.
choose your host wisely
Do you have any particular recommendations for a host?
I can tell you it will be the most expensive on AWS or Azure. That’s about all I could say without pricing stuff out.
I’d look into Hetzner, their pricing is pretty fair and they have some nifty features.
Also check out Vultr, they have block storage and some interesting addons.
That’s where I’d start, but I haven’t needed to host anything like Lemmy.
- data ingress/egress
- storage (block and DB)
Do you have any estimations on the relationship between user count and average data transfer rates, and the average rate of storage increase?
It would depend completely on how many users you let in and what kind of things they’ll be doing. Some users are super heavy with uploading images, some users aren’t.
I haven’t read the docs in a long time, but perhaps you could restrict image uploading or something. Nothing you can do about unbounded DB growth without expiring content though.
