Issue
Anyone can help explaining why sed and tee when used separately seems to behave like stream but not when combined?
You can see how stdout is progressively being streamed to stdout.
But when used together you can see stdout only at the end of the execution of the main process in the pipe.
Examples:
Note:
perl -pe "system 'sleep .03'"
is just for throttling stdout to see final stdout progressively building up
and
sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g'
is for stripping out colours
printf "
first line
\e[32msecond\e[0m line
\e[33mthird\e[0m line
\e[35mfourth\e[0m line
\e[31mfifth\e[0m line
last line
" | perl -pe "system 'sleep .03'" | sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g'
^^^ in above example you can clearly see that stdout is building progressively
printf "
first line
\e[32msecond\e[0m line
\e[33mthird\e[0m line
\e[35mfourth\e[0m line
\e[31mfifth\e[0m line
last line
" | perl -pe "system 'sleep .03'" | tee color.txt
^^^ using just tee is fine too
printf "
first line
\e[32msecond\e[0m line
\e[33mthird\e[0m line
\e[35mfourth\e[0m line
\e[31mfifth\e[0m line
last line
" | perl -pe "system 'sleep .03'" | sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g' | tee color.txt
^^^ but when combined there is clearly buffering in play.
What's causing that? If anyone can explain please?
Edited:
when sed with tee swapped it is not buffering anymore, but obviously is not what I want
printf "
first line
\e[32msecond\e[0m line
\e[33mthird\e[0m line
\e[35mfourth\e[0m line
\e[31mfifth\e[0m line
last line
" | perl -pe "system 'sleep .03'" | tee color.txt | sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g'
All of that behaves exactly the same on:
mac with zsh
mac with bash
docker run -it ubuntu bash
docker run -it ubuntu sh
$ sw_vers ProductName: macOS ProductVersion: 11.6 BuildVersion: 20G165 $ /bin/zsh --version zsh 5.8 (x86_64-apple-darwin20.0) $ /bin/bash --version GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin20) Copyright (C) 2007 Free Software Foundation, Inc.
Solution
stdout is usually line buffered only if connected to a terminal. Connecting it to a pipe causes full buffering.
Although some commands, including tee
, are guaranteed not to buffer at all (hence the last example of tee | sed
).
You can use stdbuf -oL sed 's...g'
or unbuffer sed 's...g'
to get line buffering. stdbuf
is part of GNU coreutils so should be available on Linux, It's also available by default on FreeBSD.
This article from 2006 is old, but provides some good detail. The author is a GNU coreutils maintainer.
http://www.pixelbeat.org/programming/stdio_buffering/
Answered By - dan Answer Checked By - Gilberto Lyons (WPSolving Admin)