Wednesday, December 29, 2021

[SOLVED] grep only files generated in a particular hour

Issue

I am trying to grep some pattern in a file set under a folder like below Where on the output I have to perform remaining operation. The output main.log is coming so huge almost 50k lines ,as the files starting with server02.log are almost 30 to 40 in number . The script based on this output is taking forever to complete. Is there a way that I can only take files name starting with server02.log. and generated between time 20:00:00 and 21:00:00

ls -lrth server02.log.*
-rw-r--r-- 1 user user  1.9M Apr 15 20:20 server02.log.2020
-rw-r--r-- 1 user user  1.7M Apr 15 20:30 server02.log.2030
-rw-r--r-- 1 user user  1.6M Apr 15 20:41 server02.log.2041
-rw-r--r-- 1 user user  1.9M Apr 15 20:50 server02.log.2050
-rw-r--r-- 1 user user  2.1M Apr 15 21:00 server02.log.2100
-rw-r--r-- 1 user user  1.4M Apr 15 21:10 server02.log.2110
-rw-r--r-- 1 user user  1.9M Apr 15 21:20 server02.log.2120
-rw-r--r-- 1 user user  656K Apr 15 21:29 server02.log.2129
-rw-r--r-- 1 user user  4.6M Apr 15 21:40 server02.log.2140
-rw-r--r-- 1 user user  1.9M Apr 15 21:50 server02.log.2150
-rw-r--r-- 1 user user  1.7M Apr 15 21:59 server02.log.2159
-rw-r--r-- 1 user user  724K Apr 15 22:09 server02.log.2209
-rw-r--r-- 1 user user  1.3M Apr 15 22:20 server02.log.2220
-rw-r--r-- 1 user user  1.1M Apr 15 22:29 server02.log.2229
-rw-r--r-- 1 user user  1.7M Apr 15 22:41 server02.log.2241
-rw-r--r-- 1 user user  1.5M Apr 15 22:49 server02.log.2249
-rw-r--r-- 1 user user  2.4M Apr 15 23:01 server02.log.2301
-rw-r--r-- 1 user user  1.4M Apr 15 23:10 server02.log.2310
-rw-r--r-- 1 user user  585K Apr 15 23:19 server02.log.2319
-rw-r--r-- 1 user user  858K Apr 15 23:30 server02.log.2330
-rw-r--r-- 1 user user  892K Apr 15 23:40 server02.log.2340
-rw-r--r-- 1 user user  698K Apr 15 23:49 server02.log.2349

grep -E "###Update |###Initiate |###Re-Initiate " server02.log.* >> main.log

from the comments I made the change to my code as below

#!/bin/bash

DIR="."
d=$(date +%Y-%m-%d);
log_dir="logs/$d"
PREFIX="$log_dir/srv_02.log"
#PREFIX="srv_02.log"
echo "prefix value is $PREFIX"

START_HOUR="06"

for F in "$( find "$DIR" -name "${PREFIX}*" -printf '%Tc %p\n' | grep "\ ${START_HOUR}:" )"; do
  echo "F value is $F"
grep -E "###Update |###Initiate |###Re-Initiate" "$F" >> main.log

done

error:

prefix value is logs/2021-04-16/srv_02.log

find: warning: Unix filenames usually don't contain slashes (though pathnames do).  That means that '-name `logs/2021-04-16/srv_02.log*'' will probably evaluate to false all the time on this system.  You might find the '-wholename' test more useful, or perhaps '-samefile'.  Alternatively, if you are using GNU grep, you could use 'find ... -print0 | grep -FzZ `logs/2021-04-16/osbpd_srv_02.log*''.
F value is
grep: : No such file or directory

Solution

This solution looks for files in the given directory, created during the specified hour with names matching the given prefix.

#!/bin/bash

d=$(date +%Y-%m-%d)
DIR="logs/$d/$log_dir"
PREFIX="srv_02.log"
#PREFIX=server02.log
echo "prefix value is $PREFIX"

START_HOUR="06"
for F in "$( find "$DIR" -name "${PREFIX}*" -printf '%TY-%Tm-%Td\n' | grep "\ ${START_HOUR}:" )"; do
  echo "$F"
  # grep -E "###Update |###Initiate |###Re-Initiate Assignment Milestone|###Complete Assignment Milestone|###Cancel Assignment Milestone|###Suspend Assignment Milestone|###Resume Assignment Milestone" "$F" >> main.log
done


Answered By - Mike Slinn
at