feat(log-collector-script): journal log with precise timestamp #2151
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
guessi
commented
Feb 12, 2025
| readonly JOURNALCTL_CMD='/usr/bin/journalctl --output=short-iso-precise' | ||
|
|
||
| # Collect log start from 10 days ago, could be '-10d (10 days ago)', '-10h (10 hours ago)', '-10m (10 minutes ago)' | ||
| readonly JOURNALCTL_SINCE='-10d' |
There was a problem hiding this comment.
If we need to gather log since 30 days ago, it could easily be done by setting the value as -30d.
| readonly PROGRAM_SOURCE="https://github.com/awslabs/amazon-eks-ami/blob/main/log-collector-script/" | ||
| readonly PROGRAM_NAME="$(basename "$0" .sh)" | ||
| readonly PROGRAM_DIR="/opt/log-collector" | ||
| readonly LOG_DIR="/var/log" | ||
| readonly COLLECT_DIR="/tmp/eks-log-collector" | ||
| readonly CURRENT_TIME=$(date --utc +%Y-%m-%d_%H%M-%Z) | ||
| readonly DAYS_10=$(date -d "-10 days" '+%Y-%m-%d %H:%M') |
There was a problem hiding this comment.
Before change, it is confused to have hard-coded DAYS_10 here. It's just a variable, would better to have no explicitly days defined in variable name.
|
|
||
| timeout 75 journalctl --unit=nodeadm-run --since "${DAYS_10}" > "${COLLECT_DIR}"/nodeadm/nodeadm-run.log | ||
| timeout 75 ${JOURNALCTL_CMD} --unit=nodeadm-config > "${COLLECT_DIR}"/nodeadm/nodeadm-config.log | ||
| timeout 75 ${JOURNALCTL_CMD} --unit=nodeadm-run > "${COLLECT_DIR}"/nodeadm/nodeadm-run.log |
There was a problem hiding this comment.
No need to add --since for the nodadm-{config,run} services.
guessi
force-pushed
the
fine-tune-log-collector
branch
from
e909a99 to
d275801
Compare
guessi
commented
Feb 12, 2025
| @@ -780,7 +788,7 @@ get_containerd_info() { | |||
| # force containerd to dump goroutines | |||
| timeout 75 killall -sUSR1 containerd | |||
| timeout 75 containerd config dump > "${COLLECT_DIR}"/containerd/containerd-config.txt 2>&1 || echo -e "\tTimed out, ignoring \"containerd info output \" " | |||
| timeout 75 journalctl -u containerd > "${COLLECT_DIR}"/containerd/containerd-log.txt 2>&1 || echo -e "\tTimed out, ignoring \"containerd info output \" " | |||
| timeout 75 ${JOURNALCTL_CMD} --unit=containerd --since "${JOURNALCTL_SINCE}" > "${COLLECT_DIR}"/containerd/containerd-log.txt 2>&1 || echo -e "\tTimed out, ignoring \"containerd info output \" " | |||
There was a problem hiding this comment.
Add --since flag to aligned with log gathering command for docker.
guessi
commented
Feb 12, 2025
|
|
||
| # Unify the out output with precise ISO timestamp (e.g. 2025-02-12T06:25:29.149489+0000) | ||
| # It might be needed to have nanosecond timestamp for advanced troubleshooting. | ||
| readonly JOURNALCTL_CMD='/usr/bin/journalctl --output=short-iso-precise' |
There was a problem hiding this comment.
EKS node might have customized timezone set, precisely log the timestamp would be very helpful for troubleshooting.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes:
Feb 12 06:25:29timezoneinfo (it might have different timezone setup at node bootstrap).secondonly, no that precise.2025-02-12T06:25:29.149489+0000timezoneinfo included.nanosecond.journalctlcommands inside log-collector script.${JOURNALCTL_CMD}defined, try to unify the command with--outputflag set.timeout->journalctl->--unit-->--since (if needed)--since $(date -d "-10 days" '+%Y-%m-%d %H:%M')--since "-10d"Testing Done