[SOLVED] sed and tee when working together in pipes no longer behaves like stream

Issue

Anyone can help explaining why sed and tee when used separately seems to behave like stream but not when combined?

You can see how stdout is progressively being streamed to stdout.

But when used together you can see stdout only at the end of the execution of the main process in the pipe.

Examples:

Note:

perl -pe "system 'sleep .03'"

is just for throttling stdout to see final stdout progressively building up

and

sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g'

is for stripping out colours

printf "

first line
\e[32msecond\e[0m line
\e[33mthird\e[0m line
\e[35mfourth\e[0m line
\e[31mfifth\e[0m line
last line

" | perl -pe "system 'sleep .03'" | sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g'

^^^ in above example you can clearly see that stdout is building progressively

printf "

first line
\e[32msecond\e[0m line
\e[33mthird\e[0m line
\e[35mfourth\e[0m line
\e[31mfifth\e[0m line
last line

" | perl -pe "system 'sleep .03'" | tee color.txt

^^^ using just tee is fine too

printf "

first line
\e[32msecond\e[0m line
\e[33mthird\e[0m line
\e[35mfourth\e[0m line
\e[31mfifth\e[0m line
last line

" | perl -pe "system 'sleep .03'" | sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g' | tee color.txt

^^^ but when combined there is clearly buffering in play.

What's causing that? If anyone can explain please?

Edited:

when sed with tee swapped it is not buffering anymore, but obviously is not what I want

printf "

first line
\e[32msecond\e[0m line
\e[33mthird\e[0m line
\e[35mfourth\e[0m line
\e[31mfifth\e[0m line
last line

" | perl -pe "system 'sleep .03'"  | tee color.txt | sed $'s,\x1b\\[[0-9;]*[a-zA-Z],,g'

All of that behaves exactly the same on:

• mac with zsh

• mac with bash

• docker run -it ubuntu bash

• docker run -it ubuntu sh

    $ sw_vers
  
    ProductName:    macOS
    ProductVersion: 11.6
    BuildVersion:   20G165
  
    $ /bin/zsh --version
  
    zsh 5.8 (x86_64-apple-darwin20.0)
  
    $ /bin/bash --version
  
    GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin20)
    Copyright (C) 2007 Free Software Foundation, Inc.
  

Solution

stdout is usually line buffered only if connected to a terminal. Connecting it to a pipe causes full buffering.

Although some commands, including tee, are guaranteed not to buffer at all (hence the last example of tee | sed).

You can use stdbuf -oL sed 's...g' or unbuffer sed 's...g' to get line buffering. stdbuf is part of GNU coreutils so should be available on Linux, It's also available by default on FreeBSD.

This article from 2006 is old, but provides some good detail. The author is a GNU coreutils maintainer.

http://www.pixelbeat.org/programming/stdio_buffering/

Answered By - dan

Answer Checked By - Gilberto Lyons (WPSolving Admin)