Gb1UKWU9

1Posted byu/ktsaou
2netdata Developer10 months ago1
3netdata, the open-source, real-time performance and health monitoring, released v1.7.0
4Hi,
5
6I just released netdata v1.7
7
8Check the release notes: https://github.com/firehol/netdata/releases
9
10Enjoy real-time performance and health monitoring!
11
12Costa
13
1470 Comments
15Share
16
1795% Upvoted
18This thread is archived
19New comments cannot be posted and votes cannot be cast
20SORT BY
21BEST
22
23
24eraser_6776
25IT Director/Lead Ninja
2621 points
27·
2810 months ago
29Whoa. We've been looking for something like this for Azure, one of my senior admins just had kittens when he saw the dashboard. Nicely done!
30
31You expand to support Windows and network devices with that interface and it would be huuge.
32
33Share
34Report
35Save
36Give gold
37
38
39p_lett
403 points
41·
4210 months ago
43·
44edited 10 months ago
45You expand to support Windows and network devices with that interface and it would be huuge.
46
47No Windows yet, but if you network device supports SNMP then it's in. https://github.com/firehol/netdata/blob/master/conf.d/node.d/snmp.conf.md
48
49Edit: Oh, or IPMI https://github.com/firehol/netdata/wiki/monitoring-IPMI
50
51Share
52Report
53Save
54Give gold
55
56
57OathOfFeanor
581 point
59·
6010 months ago
61Windows does SNMP too!
62
63Share
64Report
65Save
66Give gold
67
68
69Vortegan
709 points
71·
7210 months ago
73Nice. Will deploy today.
74
75Share
76Report
77Save
78Give gold
79
80
81i_pk_pjers_i
822x pcpartpicker.com/p/mbqGvK (16TB) Proxmox w/ Ubuntu 16.04 VM
838 points
84·
8510 months ago
86Very nice release, have some well earned gold. :D
87
88Share
89Report
90Save
91Give gold
92
93
94ktsaou
95netdata Developer
964 points
97·
9810 months ago
99nice! thanks!
100
101Share
102Report
103Save
104Give gold
105
106
107Imaginovskiy
1089 points
109·
11010 months ago
111Normally I'd say something like "this is great for it being free and all" but no this is just great in general! I mean just run a script and be able to find out performance stats in an easily digestible manner (looking at you N-Able!)..
112
113Share
114Report
115Save
116Give gold
117
118
119AccidentallyTheCable
12012 points
121·
12210 months ago
123Hi there! I actually just disabled netdata on some of our systems yesterday because of disk issues it was causing. I like the idea the app brings, but it is a huge resource sink.
124
125Yesterday; alerted inode usage was > 80%. Tracked down that the netdata cache folder had 110k inodes in its folder (80% of used inodes were in there).
126
127Weeks ago; alerted high disk io. Tracked down that netdata was thrashing the disk (presumably writing 110 thousand cachefiles every few seconds).
128
129Out of curiousity, why arent you using a flatdb or nosql, and writing stuff to 100s of files instead?
130
131Share
132Report
133Save
134Give gold
135
136
137ktsaou
138netdata Developer
13928 points
140·
14110 months ago
142·
143edited 10 months ago
144Hi,
145
146netdata does not read/write files while it runs, except when memory mode = map. So, for all modes except map, netdata is only writing its access.log, which you can disable. Of course it saves and loads back the database on restart.
147
148For memory mode = map, writing is controlled by the kernel, it is like swap memory (mmap'd files). The wiki gives sysctl settings for this mode, to lower the pressure on disks.
149
150110k inodes are a lot. netdata uses 1 inode per metric and 2 inodes per chart. This means you had something like 35k metrics monitored.
151
152It sounds like you had this issue on a central netdata server, not at the ones you installed on all your servers. If this is the case, it requires some tuning. Check this server: https://build.my-netdata.io - it is the central netdata of 25 hosts. Check the disk. It is idle. But you need to follow the guide.
153
154If it still gives you trouble, open a github issue. netdata should not behave like that.
155
156Share
157Report
158Save
159Give gold
160
161
162mumblerit
163Sr. Sysadmin
1647 points
165·
16610 months ago
167by default im pretty sure it stores all stats in memory, deployed this recently.
168
169Share
170Report
171Save
172Give gold
173
174
175lordpuddingcup
1765 points
177·
17810 months ago
179Damn nice release
180
181Share
182Report
183Save
184Give gold
185
186
187czechsys
1884 points
189·
19010 months ago
191I wish, there will be some authentication because having proxy/firewall on every machine is...crazy.
192
193Share
194Report
195Save
196Give gold
197
198
199ktsaou
200netdata Developer
20111 points
202·
20310 months ago
204You can use a central nginx to proxy all your netdata with authentication. The wiki has examples.
205
206Share
207Report
208Save
209Give gold
210
211
212jjasghar
21312 points
214·
21510 months ago
216Is...iptables crazy? Wait, yes, yes it is. Carry on.
217
218Share
219Report
220Save
221Give gold
222
223
224Pablohn
2253 points
226·
22710 months ago
228Just make a ssh tunnel (with -L) to access without a proxy
229
230Share
231Report
232Save
233Give gold
234
235
236wildcarde815
2372 points
238·
23910 months ago
240Manage the system firewalls with your orchestration system of choice.
241
242Share
243Report
244Save
245Give gold
246
247
248SystemWhisperer
2491 point
250·
25110 months ago
252Since it sounds like you've solved this problem, what is your firewall-capable orchestration system of choice? Does it deal well with multiple distros/releases and include a strategy for managing exceptions for the applications running on particular hosts?
253
254Share
255Report
256Save
257Give gold
258
259
260p_lett
2613 points
262·
26310 months ago
264I'm not the person you were replying to, but I use Ansible to drive ferm on my boxes. Ferm is packaged in Debian, Ubuntu (and RHEL/CentOS if you enable EPEL), which is everywhere I need to run things.
265
266My base role for ferm creates a one line config file containing
267
268@include 'ferm.d/';
269and drops files in that dir which set up a default firewall policy.
270
271Then individual roles which set up services depend on the ferm role in their meta file, drop their own files in ferm.d and notify a handler which reloads them, like this task which is part of my nginx role for a publicly accessible web server:
272
273- name: open web ports to world
274  copy:
275    dest: /etc/ferm/ferm.d/20-nginx
276    content: domain (ip ip6) chain INPUT proto tcp dport (80 443) ACCEPT;
277    validate: ferm -n %s
278  notify: reload ferm
279Exceptions are handled in the roles for the services, in less trivial cases than the nginx example I template the ferm.d file with lists of allowed IP addresses, etc.
280
281My roles aren't published anywhere, but I'd be happy to go into more detail if it's useful.
282
283Share
284Report
285Save
286Give gold
287
288
289SystemWhisperer
2901 point
291·
29210 months ago
293That looks like a useful tool. Just to be clear (because it wasn't clear to me from skimming the docs), if for example ferm.d/10-baseline and ferm.d/20-nginx both contain table filter chain INPUT blocks, the two blocks are effectively merged into INPUT in filename order, and likewise with multiple files containing DNAT rules? That'd be nice.
294
295Still, it'd be better (or at least much easier from the installer's perspective) for the app to handle its own authentication / access controls, especially in a brownfield environment.
296
297Share
298Report
299Save
300Give gold
301
302
303p_lett
3041 point
305·
30610 months ago
307Yes, exactly that. The two INPUT entries would both appear in the iptables INPUT chain in the order they are found in the config files. DNAT would work the same way.
308
309Ferm can be seen as a nicer syntax for writing iptables rules - there is a 1:1 mapping between ferm config lines and iptables rules, with the exception that ferm entries with bracketed terms expand out to multiple iptables rules. Ferm has its own system of variables and macros but I've never used them, I use ansible templates to do the same thing.
310
311Any config management driven firewall generator will work better in a greenfield setup where you can set a default deny policy from day 1 and then just add allow rules for each service running on that host, but I have retrofitted ferm onto existing systems where not everything is managed by ansible - I manually create a ferm.d/20-manualconfig which contains allow rules for the things that ansible doesn't know about.
312
313Share
314Report
315Save
316Give gold
317
318
319wildcarde815
3201 point
321·
32210 months ago
323·
324edited 10 months ago
325I use puppet and for our setup which is relatively flat a module known as 'hieratic' that allows me to specify firewall rules as part of my configuration hierarchy (hiera files). So the common file has our default allow/block setup. Then you can climb up into distro settings, host group settings, and individual node settings. The one problem right now is it goes nuclear on docker configuration bits and pieces which I need to fix. One of our distro specific rules rips out firewalld and installs iptables services in it's place so that we can just automate iptables directly. Works just as well in centos 6, ubuntu 14, and ubuntu 16. The underlying module just uses 'create_resources' commands so it's relatively agnostic unlike some other modules.
326
327edit: I'd like to get some method of chaining together profiles a little better but host groups tends to get me where i need to go in our environment currently.
328
329Share
330Report
331Save
332Give gold
333
334
335dalik
3362 points
337·
33810 months ago
339I'm really interested in a performance and health monitoring system but I do have one key requirement. The ability to view a time frame which brings all data into a single view which I can click on each stat to get more details.
340
341Example
342
343I'm looking at a dashboard with 5 severs and I notice one of the servers has high CPU that lasts for 10 seconds. I should be able to click the CPU spike or grab period of time around that 10 second spike which pulls together the RAM, the disk I/O, Network utilization, event logs, process list and resources used. The CPU utilization is likely just the symptom but in order to determine the root cause (likely an application) I need to see that period of time and have it paused for analysis. It would be great to be able to see other servers that could be related to the problem so you can jump from one server to another using that same time frame.
344
345Example 2
346
347High network traffic shown on a server. I click the graph at the point of the extended spike and I get detailed information regarding the network interface (active connections etc). I should be able to click something that brings up relevant details such as a list of process's that contribute to that burst of network activity. If I click on the process that is causing the spike, I should be able to see detailed view from the point of view of that process.
348
349Is this something netdata can do or plans to implement?
350
351Share
352Report
353Save
354Give gold
355
356
357ktsaou
358netdata Developer
3595 points
360·
36110 months ago
362Hi,
363
364Example 1 is supported by netdata already: Pan and zoom a chart on server A and then select server B from the my-netdata menu. The new server comes with exactly the same timeframe, and if the same chart exists on server B, it will also scroll to the same chart.
365
366Example 2 is somewhat supported. When you pan and zoom a chart on a netdata dashboard, all the other charts follow too. Then just scroll the page. This is an ideal way to find what influences or is influenced by the spike in question.
367
368Share
369Report
370Save
371Give gold
372
373
374dalik
3751 point
376·
37710 months ago
378Thank you, I will check it out.
379
380Share
381Report
382Save
383Give gold
384
385
386ktsaou
387netdata Developer
3882 points
389·
39010 months ago
391ok. When you try it, pan and zoom with the mouse wheel while pressing SHIFT. A lot faster and better compared to the buttons.
392
393Share
394Report
395Save
396Give gold
397
398
399mspinit
400Broad Practice Specialist
4012 points
402·
40310 months ago
404Awesome! Graphs for days
405
406Share
407Report
408Save
409Give gold
410
411
412remote_bro
4132 points
414·
41510 months ago
416I was just thinking about to put this on few dedicated high performance boxes, so that I do not have to add it to my general monitoring system. Here you are.
417
418Share
419Report
420Save
421Give gold
422
423
424Matvalicious
425SCCM Admin
4262 points
427·
42810 months ago
429Highly recommended. I run this on my home built NAS/Plex server.
430
431Share
432Report
433Save
434Give gold
435
436
437j1nx_to
4381 point
439·
44010 months ago
441Nice! Will try it out later
442
443Share
444Report
445Save
446Give gold
447
448
449Ron_Swanson_Jr
4501 point
451·
45210 months ago
453How can I get this running on ~300+ servers with a single unified dashboard?
454
455Share
456Report
457Save
458Give gold
459
460
461ktsaou
462netdata Developer
4633 points
464·
46510 months ago
466All the netdata installations you own are one. Install 2 servers and then use the my-netdata menu. All settings are propagated from server to server, including current timeframe viewed, dashboard themes, current section / chart viewed, etc.
467
468So, all your netdata are just one application.
469
470You can create custom dashboards quite easily (a div per chart, no javascript required) to build multi-server views, like the netdata home page at https://my-netdata.io.
471
472You can also push all metrics to graphite, opentsdb, prometheus, influxdb, etc and use grafana.
473
474v1.8 of netdata will also provide multi-server dashboards out of the box.
475
476Share
477Report
478Save
479Give gold
480
481
482hogie48
4831 point
484·
48510 months ago
486Does Netdata support multi servers reporting back to a centralized location? I have used Netdata before (awesome product btw!), but only on small one off projects.
487
488I noticed in the release notes there is now "Central Netdata". Do you have any more info about this, or any walk through on setup for this yet?
489
490Share
491Report
492Save
493Give gold
494
495
496ktsaou
497netdata Developer
4985 points
499·
50010 months ago
501Of course. The wiki if full of them (search for: streaming) The v1.6.0 release notes give a good overview: https://github.com/firehol/netdata/releases/tag/v1.6.0
502
503configuration overview: https://github.com/firehol/netdata/wiki/Replication-Overview
504
505Share
506Report
507Save
508Give gold
509
510
511NiftyMist
5121 point
513·
51410 months ago
515I'm trying to find a breakdown of all the chart-libraries, but the wiki page seems to be unfinished. Can someone point me in the right direct? I'm looking for options of some of the available charts to add to my custom dashboards.
516
517Share
518Report
519Save
520Give gold
521
522
523ktsaou
524netdata Developer
5252 points
526·
52710 months ago
528You are right. The docs for this kind of info are not there. I suggest to do this:
529
530View this file: https://github.com/firehol/netdata/blob/master/web/dashboard.js
531
532Search on that page for: NETDATA.dataAttribute
533
534There are a few common to all charts at the top, and after that you will find whole bunches for each chart library.
535
536Share
537Report
538Save
539Give gold
540
541
542NiftyMist
5431 point
544·
54510 months ago
546Thanks! There is another problem I'm having issues with. I'm trying to display uptime in days not seconds. Is there an easy way to do this via javascript?
547
548Share
549Report
550Save
551Give gold
552
553
554ktsaou
555netdata Developer
5563 points
557·
55810 months ago
559Like this? http://london.my-netdata.io/api/v1/badge.svg?chart=system.uptime
560
561Put your hostname there and you are done.
562
563Share
564Report
565Save
566Give gold
567
568
569NiftyMist
5701 point
571·
57210 months ago
573Yes, something like this. But we want something that can go on our dashboard and be updated in real time.
574
575Share
576Report
577Save
578Give gold
579
580
581ktsaou
582netdata Developer
5833 points
584·
58510 months ago
586Check the docs: https://github.com/firehol/netdata/wiki/Generating-Badges
587
588You can use it in an embed html tag, append &refresh= to the url to make them autorefresh, change colors, labels, etc.
589
590So, you use that URL as an image. You can do the same with any metric and any alarm (alarms also change colors automatically based on status).
591
592Share
593Report
594Save
595Give gold
596
597
598NiftyMist
5992 points
600·
60110 months ago
602Thanks so much! This is just my first go at a custom dashboard, but my boss is extremely impressed so far!
603
604My Dashboard_V1
605
606Share
607Report
608Save
609Give gold
610
611
612ktsaou
613netdata Developer
6141 point
615·
61610 months ago
617nice!
618
619Share
620Report
621Save
622Give gold
623
624
625lifeatvt
626Master of None
6271 point
628·
62910 months ago
630Anyway to scan an entire network with something derivative of this and have it all in one console?
631
632Share
633Report
634Save
635Give gold
636
637
638ktsaou
639netdata Developer
6402 points
641·
64210 months ago
643netdata already does this by itself. Check the my-netdata menu found at the top left corner of all netdata dashboards.