zMqs0bZv

1
2The topic is:  sex
312:17
4Topic set by html on 15 December 2019, 12:03:48
512:17
6go1dfish
7https://notabug.io/user/CEyKrDd1xyPXpWSV00MgvnZY2VJLHXgzCvhMeDwKTYA.yjSq0DyXzzhB_ZXr_DzfJgij3tXU0-3t0Q5bJAtZpj8/spaces/frontpage/hot Is the notabug.io frontpage space
812:17
9go1dfish
10or just / on notabug.io
1112:17
12guest80
13Why am I a guest now
1412:18
15guest80
16I am html
1712:18
18go1dfish
19https://nab.cx is the same backend, but points to a different frontpage space
2012:18
21guest80
22But can't use that name
2312:18
24go1dfish
25but both spaces are accessible on either domain with the full path
2612:19
27@magnora7
28I see
2912:20
30@magnora7
31no idea html, try refreshing the page and logging out and back in
3212:20
33guest80
34Thanks.
3512:21
36@magnora7
37d3rr, how do you make the .py files compile to make the new .pyc files? I think my gold fix would work after I do that
3812:21
39go1dfish
40My guess would be is you are still connected to irc as html
4112:21
42go1dfish
43or it thinks you are anyway
4412:21
45@magnora7
46yeah maybe multiple tabs, etc
4712:22
48d3rr
49try running 'make
5012:22
51d3rr
52' in r2. there might be downtime.
5312:22
54@magnora7
55just 'make' by itself? okay
5612:22
57@magnora7
58cool thanks
5912:22
60@magnora7
61here goes...
6212:23
63d3rr
64If that fails, the full procedure with definite downtime is here: https://github.com/libertysoft3/saidit#rebuild-reddit-open-source
6512:23
66@magnora7
67the last gray box right?
6812:23
69d3rr
70yep
7112:23
72@magnora7
73rebuild reddit open source
7412:23
75@magnora7
76okay cool
7712:24
78@magnora7
79yay it worked!
8012:24
81@magnora7
82first try
8312:24
84@magnora7
85now everyone has new comment highlighting, yay
8612:24
87d3rr
88?
8912:24
90go1dfish
91isn't that more processing intensive?
9212:25
93@magnora7
94probably a bit
9512:26
96@magnora7
97well hmm its working for my magnora7 account but not my other test accounts
9812:26
99go1dfish
100one of the first things I want to do once I dive back into frontend work is encrypted user config data, this would allow tracking that sort of thing clientside but stored in the cloud
10112:27
102go1dfish
103this is how i plan to track read vs unread messages
10412:28
105magnora3
106oh wait, yes it's working. The only bug is non-mods can't choose not to have highlighting, the preference option won't save for some reason, it's stuck on if you're not a mod of some sub
10712:28
108d3rr
109hmm
11012:28
111@magnora7
112sounds nice go1dfish
11312:29
114d3rr
115a minimalist backend sounds nice and cheap to operate.
11612:29
117go1dfish
118exactly :)
11912:30
120go1dfish
121i'm very heavily optimized towards read traffic more-so than write
12212:30
123go1dfish
124right now the client makes a lot of requests via web sockets, but handling them is always just a key value lookup of a string serverside
12512:31
126@magnora7
127It's this value "highlight_new_comments" that won't save to the database unless you're a mod of some sub. Maybe it's stored in some mod category in the database that doesn't exist unless you're a mod? Any ideas?
12812:31
129go1dfish
130with the api stuff I'm working on, I may be able to reduce the number of requests as well
13112:31
132d3rr
133There's probably validation for saving that preference that checks gold status too.
13412:31
135go1dfish
136how did you enable gold for everyone? I know gold features get enabled for mods in their own subs
13712:31
138go1dfish
139might have something to do with that
14012:32
141go1dfish
142ah yeah what d3rr said makes sense
14312:32
144@magnora7
145yeah I just enabled it so anyone can do it in any sub, that much is working
14612:32
147@magnora7
148but now it's just that the user preference to disable it can't be saved unless you're a mod
14912:32
150d3rr
151i've been at this for 2 or 3 years... i still feel lost in this codebase. I guess it would help if I knew python :)
15212:32
153@magnora7
154hmm so how do I fix that validation thing...
15512:33
156@magnora7
157can we just give everyone gold instead of removing the if statments?
15812:33
159@magnora7
160then the validation would work
16112:33
162d3rr
163that would involve writing data to the db, which seems more heavy handed to me, but it is also a valid approach
16412:34
165@magnora7
166I have zero idea how to access the validation code though, so...
16712:34
168d3rr
169i think there's a preferences model. you can search for the name of that checkbox.
17012:34
171go1dfish
172yeah this is part of why I didn't want to just fork reddit, the codebase seemed pretty crufty and setup difficult
17312:35
174go1dfish
175my goal is that nab should be installable with only node/yarn as dependencies and less than 3 commands
17612:35
177d3rr
178they left so many things out of the open source version, it was basically just token open source near the end
17912:35
180@magnora7
181oh wait I think I found it
18212:35
183go1dfish
184no separate database software even
18512:35
186@magnora7
187I think this is the one that's forcing that state: https://github.com/libertysoft3/saidit/blob/14c61c334053cd01630ceefaa04a20d26b50a7f5/r2/r2/lib/validator/preferences.py#L198
18812:36
189d3rr
190huge critical things like missing cron jobs, configuration that wasn't shown in the config file, etc.
19112:36
192@magnora7
193I'm just going to delete those two lines
19412:36
195go1dfish
196they treated their commitment to open source like their commitment to free speech
19712:36
198d3rr
199that sounds nice. our dependencies are gonna kill us, like freaking facebook dependencies from 2014
20012:36
201go1dfish
202at least they finally got honest about open source
20312:36
204d3rr
205hahhaa yep
20612:37
207go1dfish
208https://youtu.be/uo4O4T-7BiE?t=45
20912:37
210d3rr
211you found it! nice
21212:37
213@magnora7
214:)
21512:38
216@magnora7
217okay, running the 'make', here we go
21812:39
219@magnora7
220should be fully functional new comment highlighting for all users after this
22112:39
222Tiwaking waits
22312:39
224@magnora7
225okay all done, now to test it
22612:39
227Tiwaking
228Highlight comments posted since previous visit:
22912:40
230magnora3
231cool, seems to work on my test account too, as does the preference saving
23212:41
233@magnora7
234now I just have to modify the preferences page to make it look like a normal feature instead of a gold feature
23512:43
236guest80
237What the nigger
23812:43
239guest80
240why can’t I be u/html
24112:43
242d3rr
243it looks like the daymode color could use adjusting too. it's always that last 20% that kills you.
24412:43
245Tiwaking
246I'll try to be html
24712:44
248html
249This is Tiwaking
25012:44
251html
252OMG what???? This is Tiwaking
25312:44
254html
255I'll go back
25612:44
257Tiwaking
258What
25912:44
260Tiwaking
261html what have you done??
26212:45
263d3rr
264he'll be html again in 24h when everything resets.
26512:45
266Tiwaking
267Ahh okay. I thought he had broken something
26812:45
269d3rr
270this is v1 integration, it's not bullet proof by any means
27112:46
272d3rr
273sad video go1dfish, they used to get it.
27412:47
275guest80 I'm a negro
27612:47
277Tiwaking
278Knee....Grow?
27912:48
280guest80
281How did you change your name?
28212:48
283Tiwaking
284Click on the pencil next to your name
28512:48
286Tiwaking
287By the "Chat in all" textbox
28812:49
289d3rr
290i'm out. it's been swell. best of luck to all.
29112:49
292@magnora7
293cool got the preferences looking good, we're all
29412:49
295@magnora7
296set
29712:49
298Tiwaking
299Night d3rr! See you again some time!
30012:49
301@magnora7
302goodnight d3rr, peace
30312:49
304d3rr
305thanks guys
30612:49
307go1dfish
308gnite
30912:49
310guest80
311#itsafetish
31212:49
313guest80
314Cya
31513:07
316@magnora7
317wow saidit is under crazy attack right now
31813:07
319@magnora7
320getting DDOS'd very badly
32113:08
322go1dfish
323lame
32413:09
325@magnora7
326yeah, even with the cloudflare on they almost took down the server
32713:09
328@magnora7
329There's been about 4 separate attempts to DDOS us over the last 24 hours
33013:09
331d3rr
332yeah wtf someone is trying preetty hard
33313:10
334d3rr
335some of these guys can beat cloudflare. it's freaking rough.
33613:10
337go1dfish
338are they getting past cloudflare?
33913:10
340go1dfish
341or just finding effective attacks in spite of cloudflare?
34213:11
343d3rr
344in spite of cloudflare. last time we let our ip leak, but lately that hasn't been the issue
34513:12
346go1dfish
347you might consider looking into cloudflare rate limiting
34813:12
349d3rr
350one of them implied he can beat the CF js check by just keeping the connection open for a long time.
35113:13
352go1dfish
353but if they are truly distributed it wont help much I suppose
35413:13
355d3rr
356yes but they want big $$ for that
35713:13
358d3rr
359i think we need a DDOS friendly hosting provider, or maybe some kind of rate limiter/load balancer in front of the main server.
36013:13
361go1dfish
362yeah I was looking at it as an option for write rate limiting which is way cheaper than doing it for reads
36313:14
364d3rr
365i'm not clear on how they beat cloudflare, but they definitely have thousands of clean ips at their disposal
36613:15
367go1dfish
368yeah that's gonna be tough, if it's the same sort of attack that voat alleges that might explain why they went to requiring logins
36913:16
370d3rr
371yeah. i guess we'd do the same if it made sense and we had to.
37213:17
373go1dfish
374how much cacheing do you do on your own?
37513:18
376go1dfish
377if they are hitting reads some sort of cacheing layer might be a good way to mitigate
37813:18
379d3rr
380we haven't added much caching, but the reddit code has extensive read caching
38113:18
382d3rr
383postgres is cached in cassandra which is cached again in memcached
38413:20
385go1dfish
386are they cachebusting somehow?
38713:20
388@magnora7
389they are going through about 4,000 IP addresses per hour according to the stats, which is probably how they're even getting past cloudflare to a minor extent
39013:21
391go1dfish
392damn
39313:21
394d3rr
395yeah it wouldnt surprise me if they are cachebusting. i haven't dug into the traffic too much
39613:21
397go1dfish
398so if it were me, this is the approach I'd explore...
39913:21
400go1dfish
401cache extremely aggressively for non logged in accounts
40213:22
403go1dfish
404disable any really intensive endpoints for guests as well
40513:22
406@magnora7
407it looks like the last burst was a high-data low-number-of-requests type of attack, unlike the last one which was high-request on unchached data
40813:22
409@magnora7
410they're trying different weak points
41113:22
412go1dfish
413well on the bright side at least they are helping you pinpoint them heh
41413:22
415@magnora7
416this time the % cached is staying high, unlike a the attack a many hours ago where it dropped dramatically
41713:23
418go1dfish
419this is how I look at fuckery on nab
42013:23
421@magnora7
422yeah and I mean they're giving us tons of traffic which just boosts our alexa ranking too lol
42313:23
424d3rr
425our nginx rate limiting is denying tons of requests,... pretty sketchy right now though
42613:23
427go1dfish
428I think alexa is toolbar based so not likely to affect rank
42913:23
430@magnora7
431the irc is staying up since it's on another server, that's nice too
43213:23
433Tiwaking cheers
43413:24
435go1dfish
436so you can probably adjust maximum request body size in nginx
43713:24
438go1dfish
439though if you support image uploads or anything that could be problematic
44013:24
441d3rr
442that's a good one, i haven't set that
44313:24
444@magnora7
445either that or they're targeting images
44613:25
447d3rr
448our captchas are newly enabled too.. it could be that
44913:25
450@magnora7
451the requests haven't budged this time, despite the unique visitor number quadrupling
45213:25
453@magnora7
454someone is wasting effort and money lol
45513:26
456d3rr
457im gonna start blocking ips or we're done for
45813:26
459go1dfish
460are the requests gets or something else?
46113:27
462@magnora7
463we're doing okay
46413:27
465go1dfish
466if they are something else you might consider a lockdown mode for guests that only allows gets
46713:27
468@magnora7
469it's a little dicey but we're fine
47013:27
471@magnora7
472that's a good idea
47313:27
474go1dfish
475search is probably a killer as well
47613:28
477@magnora7
478maybe yeah
47913:28
480go1dfish
481if it were me I'd really not want to go login only like voat, but I"d be very open to disabling expensive features for guests
48213:28
483d3rr
484theyre all GET it appears
48513:28
486@magnora7
487good to know, thanks d3rr
48813:28
489go1dfish
490then the request body limit might be your best bet
49113:28
492@magnora7
493yeah we can have stages of locked-down-ness based on what we need at the moment
49413:29
495d3rr
496were not doing good, 30s page load times
49713:29
498@magnora7
499yeah it's chunky but mostly functional
50013:29
501go1dfish
502yeah site was real slow last I tried loading a page, irc works well though heh
50313:29
504@magnora7
505yeah irc is lightning fast lol
50613:30
507@magnora7
508there, pageloads seem good again
50913:30
510d3rr
511okay load times have recovered, nginx is no longer rate limiting so much, i think they stopped
51213:30
513@magnora7
514yeah they must've
51513:30
516@magnora7
517seems like they try an attack every 4-5 hours, going by the data
51813:30
519@magnora7
520so we'll just plan for that
52113:31
522@magnora7
523actually I guess every 8-12 hours
52413:31
525go1dfish
526this seems like it would be a lame time to attack
52713:31
528go1dfish
529if this site is at all like reddit peak traffic is american 9-5
53013:32
531@magnora7
532we've got a good chunk of european and aus and HK traffic
53313:32
534go1dfish
535that's cool
53613:33
537@magnora7
538but yeah who knows what they're thinking lol
53913:34
540@magnora7
541The numbers say about 60% of our traffic is from the US and the rest is international
54213:35
543d3rr
544i spoke to soon... blocking ips...
54513:40
546d3rr
547blocking 23 ips has saved 3800 requests.  hopefully im putting a dent in it
54813:41
549Tiwaking
550Thats good ?
55113:42
552go1dfish
553here's an idea for a nuclear option....
55413:42
555go1dfish
556im assuming these get requests have query strings?
55713:42
558go1dfish
559throw an error page for guest requests using query strings
56013:43
561Tiwaking
562Ouu thats smart!!!
56313:43
564go1dfish
565that would break going to the next page
56613:43
567go1dfish
568but if the error message is descriptive or a login page that's at least marginally better than being down entirely or requiring login in general
56913:45
570d3rr
571Somehow they requested this url? Fuuuh https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=18&ved=2ahUKEwj_ocXGoP7lAhUConEKHZoiCUUQFjARegQIBBAB&url=https%3A%2F%2Fsecurity.stackexchange.com%2Fquestions%2F57419%2Fwhat-risks-are-introduced-by-the-browser-passing-the-referer-to-the-next-site&usg=AOvVaw1rQ9S7xfXPOKGpOYeqfyKW
57213:46
573Tiwaking
574What on earth
57513:46
576go1dfish
577hmm I would think nginx would just drop that pretty quickly
57813:47
579go1dfish
580@d3rr you may want to investigate https://www.fail2ban.org
58113:47
582Tiwaking
583https://security.stackexchange.com/questions/57419/what-risks-are-introduced-by-the-browser-passing-the-referer-to-the-next-site
58413:47
585d3rr
586oh, our nginx doesn't say our domain name, good call
58713:48
588d3rr
589yeah we've got fail2ban for ssh access
59013:48
591d3rr
592and 80/443 are only allowed from cloudflare
59313:49
594d3rr
595ugh. it's the homepage, they are DDOSing the homepage with GETs
59613:49
597go1dfish
598the homepage isn't memcached?
59913:49
600d3rr
601i believe it all is aside from captcha
60213:54
603d3rr
604thanks for the help goldfish. heck of a way to spend a saturday.
60513:55
606go1dfish
607np
60814:03
609@magnora7
610oh here we go again...
61114:04
612@magnora7
613yeah cloudflare is now showing the traffic from the last attacks
61414:04
615@magnora7
616almost 1 million requests in an hour
61714:04
618Tiwaking
619wtf
62014:04
621@magnora7
622using about 7k IP addresses
62314:04
624@magnora7
625yeah wtf indeed
62614:04
627Tiwaking
628Thats a Chinese Cannon scale attack
62914:05
630@magnora7
631yeah it's a big attack for sure
63214:05
633go1dfish
634yeah that's pretty crazy
63514:06
636Tiwaking
637Are they testing it on you guys? They used it against HK already
63814:06
639Tiwaking
640You said that 60% of the traffic is American so that cant be China
64114:06
642@magnora7
643we don't know where the attacks are from because they're using VPN and changing IPs
64414:07
645@magnora7
646so really we have no clue
64714:07
648go1dfish
649That 60% number was referring to normal traffic I think
65014:07
651@magnora7
652yeah normal traffic
65314:07
654Tiwaking
655So it could be the Great Cannon of China
65614:08
657@magnora7
658who knows, it's probably more likely some cam streamer trying to shut down IP2 with a hired attack, we've had attacks like that happen more than once in just the last few months
65914:09
660Tiwaking
661Ahh okay. I always forget about those guys because I muted them
66214:09
663d3rr
664IP2 has been posting about it too, maybe that encouraged a repeat attack
66514:09
666@magnora7
667and IP2 was being trolled very heavily today as well, we had to IP ban a few users because of it
66814:09
669@magnora7
670yeah I think that's the likely culprit
67114:09
672d3rr
673don't worry guys, i have 100 ips blocked, 6900 more to go :)
67414:09
675@magnora7
676hahahaha
67714:10
678@magnora7
679are you blocking ranges of IPs?
68014:11
681@magnora7
682I'm not even sure it's worth blocking them, they just use an IP for 30 seconds and then never use it again maybe
68314:11
684@magnora7
685or maybe they cycle back to re-use them, I don't know
68614:11
687@magnora7
688IPv6 has a very large number of possibilities
68914:14
690d3rr
691nope, individual ips. i have prevented 20k requests with it so far.
69214:15
693@magnora7
694oh awesome, that's great
69514:15
696@magnora7
697nice work
69814:15
699d3rr
700you can see it in the cloudflare firewall rules. but i think youre right, they just dump the ip and use another.
70114:15
702d3rr
703thanks
70414:17
705@magnora7
706I sure hope no innocent users who just are refreshing the page a lot who get removed
70714:17
708@magnora7
709or are you only taking out ones that make crazy numbers of requests or something
71014:19
711d3rr
712only if nginx returned a 429 rate limit denied, then i paste that into CF. a few innocent users might have got banned
71314:19
714d3rr
715429 as seen in access.log, sudo tail -f /var/log/nginx/access.log | grep 429
71614:23
717@magnora7
718got it
71914:26
720Tiwaking
721When free speech is outlawed, then there are no innocent users
72214:28
723d3rr
724whew things seem to have calmed down. i guess we won.
72514:29
726d3rr
727damnit. they are watching this chat i think.
72814:30
729guest80
730Crossed pretty hard
73114:30
732guest80
733*ddoses
73414:30
735guest80
736*ddossed
73714:31
738guest80
739Its definitely not the Chinese cannon
74014:31
741Tiwaking
742html hurry up and log back in as yourself
74314:31
744d3rr
745it's less embarrassing than when legit traffic would take us down at least :)
74614:31
747guest80
748I did but I still can't be myself
74914:32
750guest80
751Someone most likely hired another person to DDoS this site
75214:32
753Tiwaking
754Then at least change your nick to html
75514:33
756guest80
757 I can't I'm on mobile
75814:34
759Tiwaking
760Ahh I see
76114:36
762go1dfish
763that sounds like something you could automate with fail2ban
76414:36
765go1dfish
766though the cloudflare bit complicates things
76714:40
768@magnora7
769yeah they're definitely watching chat, every time you say you're leaving it kicks up again
77014:40
771d3rr
772maybe if we had a ddos box in front of the main server, then it would get tons of CPU for nginx to do blocks
77314:40
774d3rr
775yeah
77614:41
777@magnora7
778is the cpu use by nginx to do the blocks a significant factor?
77914:41
780d3rr
781TFW you have mad skillz and thousands of ips, and still can't take down a backwoods reddit clone ?
78214:41
783d3rr
784i think so, when we are getting peak attack
78514:41
786d3rr
787i donno though, it should be
78814:42
789d3rr
790shouldn't be*
79114:43
792d3rr
793i will look into fail2ban automating this stuff.
79414:44
795guest4327
796beef
79714:45
798go1dfish
799due to cloudflare you can't just add a firewall rule, you'd have to use some sort of cloudflare api to tell them to block, or have nginx do the blocking based on headers
80014:46
801@magnora7
802cool cool, sounds good
80314:46
804go1dfish
805if nginx is already rejecting them with 429's though then I'm not sure if another nginx block will help much beyond that
80614:47
807guest4327
808crab-apples?
80914:48
810d3rr
811yes nginx is rejecting with 429s but it seems like they are still overwhelming us. not use if it's the 429 traffic, or other traffic that's getting through to the app. probably the latter.
81214:49
813d3rr
814How much $$ do you want to stop this DDOS??
81514:49
816go1dfish
817dont negotiate with terrorists :P
81814:50
819Tiwaking
820If they wanted money they would have asked you for money before the attack
82114:51
822d3rr
823just testing the waters :)
82414:52
825d3rr
826(earlier I read the logs wrong, that google url was the referrer they set, they were requesting the home page)
82714:53
828@magnora7
829ah good to know
83014:53
831go1dfish
832I’d look into the cacheing logic and what factors invalidate it
83314:53
834go1dfish
835referer may be one
83614:54
837go1dfish
838like I know reddit for quite a while has returned different results from google results
83914:55
840d3rr
841oh that's dirty. ok i will check that.
84214:55
843go1dfish
844yeah this is what I was referring to earlier with cachebusting
84514:55
846go1dfish
847query params are another way attackers will try to bypass cache which is why I suggested disabling those for guests
84814:59
849d3rr
850noted
85115:11
852d3rr has changed the topic to:  Welcome to SaidIt
85315:13
854d3rr
855That's it? I've got Gateron clear, I can go all night fucker.
85615:14
857Tiwaking
858YAY!!
85915:14
860Tiwaking is glad Saidit is safe again!
86115:14
862d3rr
863yay
86415:14
865Tiwaking
866MSSA? Make Saidit Safe Again?
86715:15
868d3rr
869hahaa
87015:15
871d3rr
872i don't even believe in safety anymore, this DDOS shit is hard to prevent