High system load caused by scheduled reports execution

[filippocarletti]

Steps to reproduce

• Run /usr/bin/ns-push-reports
• Observe the system process tree during scheduled report times
• Note multiple heavy processes spawned by reports

Expected behavior

• Reports and associated scripts (e.g., ns-push-reports) should not cause high system load for prolonged times on typical hardware

Actual behavior

• High system load observed when reports are generated

Example process tree (output from ps command):

3292 ?        S      0:17 /usr/sbin/crond -f -c /etc/crontabs -l 5
 5708 ?        S      0:00  \_ /bin/bash /usr/bin/ns-push-reports
 6365 ?        S      0:00      \_ curl --fail -sL -H Content-Type: application/json -u d4ac30b1-31da-4671-a755-6e4f5de90f93:358b80855549b77721d58a1427bc4a0ec8c1dc5b3d7dfa5bd7529a62b5080735 -o /dev/null http://172.27.114.1:20003/ingest/dump-mwan-events --data @-
 6364 ?        S      0:00      \_ /usr/bin/python3 /usr/libexec/rpcd/ns.controller call dump-mwan-events
 6369 ?        S      0:00          \_ /bin/sh -c sed -n "/$(date -d '-20 minute' +'%b %e %H:%M')/,$ p" /var/log/messages | grep -E 'mwan3track.*is online|is offline'
 6370 ?        R      0:11              \_ sed -n /Nov 20 17:25/,$ p /var/log/messages
 6371 ?        S      0:00              \_ grep -E mwan3track.*is online|is offline

• System resources are heavily used while log processing and report pushes occur

Components

On the above system, /var/log/messages was big because it had ipsec debug logs active to troubleshoot an issue.