Friday, February 18, 2022

[SOLVED] How to generate a list of all dates in a range using the tools available in bash?

Issue

I want to download a bunch of files named with ISO-8601 dates. Is there a simple way to do this using bash+GNU coreutils? (Or some trick to make wget/curl to generate the list automatically, but I find that unlikely)

Similar to this question, but not restricted to weekdays: How to generate a range of nonweekend dates using tools available in bash?. I guess that there is a simpler way to do it without that restriction.

Also related to How to generate date range for random data on bash, but not restricted to a single year.


Solution

If you have GNU date, you could do use either a for loop in any POSIX-compliant shell:

# with "for"
for i in {1..5}; do 
    # ISO 8601 (e.g. 2020-02-20) using -I
    date -I -d "2014-06-28 +$i days"

    # custom format using +
    date +%Y/%m/%d -d "2014-06-28 +$i days"
done

or an until loop, this time using Bash's extended test [[:

# with "until"
d="2014-06-29"
until [[ $d > 2014-07-03 ]]; do 
    echo "$d"
    d=$(date -I -d "$d + 1 day")
done

Note that non-ancient versions of sh will also do lexicographical comparison if you change the condition to [ "$d" \> 2014-07-03 ].

Output from either of those loops:

2014-06-29
2014-06-30
2014-07-01
2014-07-02
2014-07-03

For a more portable way to do the same thing, you could use a Perl script:

use strict;
use warnings;
use Time::Piece;
use Time::Seconds;    
use File::Fetch;

my ($t, $end) = map { Time::Piece->strptime($_, "%Y-%m-%d") } @ARGV; 

while ($t <= $end) {
    my $url = "http://www.example.com/" . $t->strftime("%F") . ".log";
    my $ff = File::Fetch->new( uri => $url );
    my $where = $ff->fetch( to => '.' );  # download to current directory
    $t += ONE_DAY;
}

Time::Piece, Time::Seconds and File::Fetch are all core modules. Use it like perl wget.pl 2014-06-29 2014-07-03.



Answered By - Tom Fenech
Answer Checked By - Candace Johnson (WPSolving Volunteer)
at