It is currently Wed, 22 Oct 2014 22:49:30 GMT



 
Author Message
 Is there any way to force line-buffering in a pipeline?

From time to time I want to set up a pipeline like this:
$ program | unix_tool | myfilter

where:
"program" receives data at irregular intervals from
a serial port and flushes lines of output to stdout.

"unix_tool" is one of the more-or-less standard unix programs
which does not have an option for line-buffering its output.

"myfilter" is a C program intended to do something with lines
of input and write them to stdout.  I've written the code
similar to this fragment:
  while (1) {
     if ( fgets(line, sizeof(line), stdin) == NULL ) {
        sleep(1);
        continue;
     }
     do_something_with_line_and_print(line);
  }

The problem of course is that the output from "unix-tool" is
block-buffered, so "myfilter" doesn't get any input until
the buffer fills and is flushed, whereas I want to see the
output as soon as the data is received.

I'm looking for a way of making this happen short of hacking the
source for "unix-tool".

I'm running the GNU bash shell under Linux.  Would some other
shell be more appropriate for this purpose?

Should "myfilter" be written differently?

Is there any way for "myfilter" to fool the shell into
thinking it's a terminal?

Thanks for your help.

Regards,
Charles Sullivan



 Thu, 21 Jun 2007 01:44:52 GMT   
 Is there any way to force line-buffering in a pipeline?
In article <pan.2005.01.01.17.44.51.805...@triad.rr.com>,
Charles Sullivan  <cwsul...@triad.rr.com> wrote:

Look for the "unbuffer" program in the Expect distribution.



 Thu, 21 Jun 2007 02:19:47 GMT   
 Is there any way to force line-buffering in a pipeline?

Thanks for the tip Kenny - it sounds exactly like what I need.
I installed 'expect' via RPM from Yum, but unfortunately I'm
not having any success in making it work.

GNU (and perhaps other) grep has a --line-buffered option, so
I compared these command lines when 'program' outputs one short
line of data containing the token shown here as "xxxx":

No output here, as expected:
$ /usr/local/bin/program | /bin/grep xxxx | /bin/grep xxxx

Output here, as expected ("program" flushes its output):
$ /usr/local/bin/program | /bin/grep --line-buffered xxxx | /bin/grep xxxx

But no output here:
$ /usr/local/bin/program | unbuffer /bin/grep xxxx | /bin grep xxxx

or here either:
$ unbuffer /usr/local/bin/program | unbuffer /bin/grep xxxx | /bin/grep
xxxx

or even here:
$ unbuffer /usr/local/bin/program | unbuffer /bin/grep xxxx

Any advice?

Regards,
Charles Sullivan



 Thu, 21 Jun 2007 05:55:48 GMT   
 Is there any way to force line-buffering in a pipeline?

I have done such task with a pty (see Stevens' book "Advanced Programming
in the Unix Environment"); you want to try that if you have problems with
unbuffer. You will find the sourcecode here...

http://www.yendor.com/programming/unix/apue/ch19.html

Janis



 Thu, 21 Jun 2007 07:24:32 GMT   
 Is there any way to force line-buffering in a pipeline?

Thanks Janis, I'll check it out.  I'm surprised that a tool like
this isn't automatically included in mainstream workstation
distros.



 Thu, 21 Jun 2007 21:08:38 GMT   
 Is there any way to force line-buffering in a pipeline?
In article <pan.2005.01.02.13.08.33.68...@triad.rr.com>,
Charles Sullivan  <cwsul...@triad.rr.com> wrote:
...

Conceptually, there's no difference between pty and Expect.  I'm surprised
that you were not able to get Expect to work for you.

Expect *is* included in most Linux distros.



 Thu, 21 Jun 2007 21:19:35 GMT   
 Is there any way to force line-buffering in a pipeline?

Indeed; the first thing I've done when I heared about pty was to try
whether it was available on any Unix platform. But since Stevens pty
was easy to compile and worked without problems I don't complain. :-)

Janis



 Thu, 21 Jun 2007 22:33:00 GMT   
 Is there any way to force line-buffering in a pipeline?

You're right!!!  I just went back and looked at my Red Hat 9 and
Fedora Core 2 distro CDs, and it's there on both of them - same
versions (different for each distro) as I installed via YUM.
But AFAIK neither was included in the packages installed during the
initial installation, even though I select all the groupings
related to workstations, whether I know what it's about or not.

I tried rebooting FC2 back to the original disto kernel and
testing "unbuffer" at the text console, but still no-go.  But
I've also upgraded libraries along the way and that may have
something to do with the problem.

This whole situation bears further looking into.  Thanks very
much for your help, Kenny and Janis.

Regards,
Charles Sullivan



 Fri, 22 Jun 2007 00:41:09 GMT   
 Is there any way to force line-buffering in a pipeline?
Charles Sullivan <cwsul...@triad.rr.com> wrote, on Sat, 01 Jan 2005:

By default unbuffer does not read from stdin.  If you have a
recent version it will accept a -p option to make it usable in
a pipeline.  However, it is awkward to use because it exits when
it sees EOF on input, so you have to resort to things like:

$ (/usr/local/bin/program; sleep 5) | unbuffer -p grep xxxx | grep xxxx

Using sleep to keep unbuffer's stdin open is quick and easy, but
not very reliable.  However, I don't think there is any reasonably
simple way to do it better, because there is a catch-22: you want
the final grep to have finished reading the data before you close
unbuffer's stdin, but the grep will not exit until it gets EOF, which
won't happen until after unbuffer gets EOF.

--
Geoff Clare <netn...@gclare.org.uk>



 Sat, 23 Jun 2007 22:16:58 GMT   
 Is there any way to force line-buffering in a pipeline?

Thanks Goeff.  The precompiled version for Fedora Core 2 doesn't
have that feature - I'll have to go to the source.



 Sun, 24 Jun 2007 13:16:13 GMT   
 
   [ 10 post ] 

Similar Threads

1. forcing line buffering?

2. Line buffer and Full buffer

3. Buffering output in a pipeline

4. I AM GONNA PUKE - I AM GONNA PUKE - I AM GONNA PUKE - I AM GONNA PUKE - I AM GONNA PUKE - I AM GONNA PUKE - I AM GONNA PUKE -

5. newbie: mouse setup: I am forced to re-plug the mouse

6. splitting (teeing) a pipeline into two further pipelines

7. force clearing buffer

8. Forcing a flush of stdout buffer

9. Want to force non-buffered read of floppy


 
Powered by phpBB © 2000, 2002, 2005, 2007 phpBB Group.
Designed by ST Software