I don’t get it. Public endpoints are public. Go proxies (there are alternatives to direct mode or using Google proxy, such as Athens) are legitimate to query these public endpoints, as aggressively as they want. That’s not polite, but that’s how the open Internet works and always has.
I don’t get why SourceHut does not have any form of DDoS protection, or rate-limiting. I mean HTTP status 503 and the retry-after header are standard HTTP. That Drew chose a public outcry over implementing basic anti-applicative DDoS seems to be a very questionnable strategy. What would happen to the Sourcehut content if tomorrow attackers launch a DDoS attack on SourceHut? Will Drew post another public outcry on their blog?
SourceHut is still in alpha. This feels like a sign that it is still not mature enough to be a prod service for anyone.
Google refuses to respect a robots.txt here, seems awful naive to assume they will respect a 503 response or retry-after header.
Similarly naive to assume there’s no ddos mitigation in front of sourcehut, given that Drew explained why he allows the proxy traffic through unabated.
To then take that naive assumption and leap off it to conclusions about the production readiness of alpha software is some wild FUD.
I don’t think that a robots.txt file is the appropriate tool here.
First off, robots.txt are just hints for respectful crawlers. Go proxies are not crawlers. They are just that: caching proxies for Go modules. If all Go developers were to use direct mode, I think the SourceHut traffic would be more, not less.
Second, let’s assume that Go devs would be willing to implement something to be mindful of robots.txt or retry-after indications. Would attackers do? Of course not.
If a legitimate although quite aggressive traffic is DDoSing SourceHut, that is primarily a SourceHut issue. Returning a 503 does not have to be respected by the client because the client has nothing to respect: the server just choose to say “I don’t want to answer that request. Good Bye”. This is certainly not a response that is costly to generate.
Now, if the server tries to honor all requests and is poorly optimized, then the fault is on the server, not the client.
I have not read in details the Go Proxy implementation, to be truthful. I don’t know how it would react if SourceHut was answering 503 status code every now and then, when the fetching strategy is too aggressive. I would simply guess that the server would retry later and serve the Go developers a stale version of the module.
(I would appreciate if the down voters were able to express their disagreement with words. Maybe I’m wrong, but then, please do me the favor of explaining me how.
Also, I’m not a SourceHut hater; I even give money to Drew every month, because I like the idea of SourceHut. I just think Drew is wrong on that matter)
I don’t get it. Public endpoints are public. Go proxies (there are alternatives to direct mode or using Google proxy, such as Athens) are legitimate to query these public endpoints, as aggressively as they want. That’s not polite, but that’s how the open Internet works and always has.
I don’t get why SourceHut does not have any form of DDoS protection, or rate-limiting. I mean HTTP status 503 and the
retry-afterheader are standard HTTP. That Drew chose a public outcry over implementing basic anti-applicative DDoS seems to be a very questionnable strategy. What would happen to the Sourcehut content if tomorrow attackers launch a DDoS attack on SourceHut? Will Drew post another public outcry on their blog?SourceHut is still in alpha. This feels like a sign that it is still not mature enough to be a prod service for anyone.
Google refuses to respect a robots.txt here, seems awful naive to assume they will respect a 503 response or retry-after header.
Similarly naive to assume there’s no ddos mitigation in front of sourcehut, given that Drew explained why he allows the proxy traffic through unabated.
To then take that naive assumption and leap off it to conclusions about the production readiness of alpha software is some wild FUD.
I don’t think that a robots.txt file is the appropriate tool here.
First off, robots.txt are just hints for respectful crawlers. Go proxies are not crawlers. They are just that: caching proxies for Go modules. If all Go developers were to use direct mode, I think the SourceHut traffic would be more, not less.
Second, let’s assume that Go devs would be willing to implement something to be mindful of robots.txt or
retry-afterindications. Would attackers do? Of course not.If a legitimate although quite aggressive traffic is DDoSing SourceHut, that is primarily a SourceHut issue. Returning a 503 does not have to be respected by the client because the client has nothing to respect: the server just choose to say “I don’t want to answer that request. Good Bye”. This is certainly not a response that is costly to generate. Now, if the server tries to honor all requests and is poorly optimized, then the fault is on the server, not the client.
I have not read in details the Go Proxy implementation, to be truthful. I don’t know how it would react if SourceHut was answering 503 status code every now and then, when the fetching strategy is too aggressive. I would simply guess that the server would retry later and serve the Go developers a stale version of the module.
(I would appreciate if the down voters were able to express their disagreement with words. Maybe I’m wrong, but then, please do me the favor of explaining me how. Also, I’m not a SourceHut hater; I even give money to Drew every month, because I like the idea of SourceHut. I just think Drew is wrong on that matter)
https://github.com/gomods/athens/blob/723c06bd8c13cc7bd238e650a559258ff7e23142/pkg/module/go_get_fetcher.go#L145-L148 https://github.com/gomods/athens/blob/723c06bd8c13cc7bd238e650a559258ff7e23142/pkg/module/go_get_fetcher.go#L163-L165
So two infos: