For a project I'm working on, I have written a kernel module that
hooks the RTC timer interrupt.  The module sets up the RTC so it will
interrupt at a rate of 8Khz (or about 122 microseconds per tick).

To get a feel for how much jitter there is associated with this
interrupt, I wrote some code to measure elapsed time between
consecutive interrupts (which should obviously be around 122
microseconds).  This measurement is taken using the rdtscl function.
The code keeps track of the maximum elapsed time and whenever a new
max is hit, it prints out a message reporting the new max.

When I ran my test, I was a suprised to see that my code measured a
maximum of over 2 milliseconds between consecutive interrupts.
Further testing showed that in most cases, my timer interrupt was
pretty much on time, but occasionally (like 10 times in 30 minutes) it
was getting whacked into the millisecond range.  The lead me to
hypothesize that somewhere in the system, somebody is shutting off
interrupts for around 2 milliseconds, which caused my code to miss a
bunch of RTC timer interrupts.

So, to prove my hypothesis, I set out to measure how long interrupts
were being shut off for.  I did this by modifying the kernel to
intercept the save_flags and restore_flags macros.  The basic idea of
my modification goes like this:

When save_flags is called, I look to see if I'm going from an
Interrupts Enabled(IE) to an Interrupts Disabled (ID) state.  If I am,
I grab the time using rdtscl and save it in a global variable.

When restore_flags is called, I look to see if I'm going from an ID
state to an IE state.  If I am, I grab the time using rdtscl.  Then I
calculate the difference between the two times.  If the difference is
greater than the max difference so far, I save the new max and call
"show_stack(0)".  I call show_stack so I can see where I am when the
new max takes place.  The idea is that this will identify the section
of code that is shutting off interrupts for long periods of time.
Armed with that knowledge, I'm hoping I can figure out how to avoid
the long delay.

And now finally, I come to my problem.  When I run my kernel module
with my hacked, interrupt-off-measuring kernel,  I see results that
don't make sense.  I see my kernel telling me that the maximum amount
of time that interrupts are shut off for is literally in the hundreds
of nanoseconds.  However, during this time, my module is telling me
that it measured a time between consecutive interrupts of 2
milliseconds!  So if nobody is shutting off interrupts for more than a
few hundred nanoseconds who the heck is holding off my timer
interrupt???

One guess is that there are other ways to shut off and turn on
interrupts.  Perhaps there is code that calls cli and sti directly?
Or, could somebody be masking all interrupts but theirs?  Or, does the
RTC hardware go out to lunch occasionally?

Thanks for taking the time to read this.  Any thoughts or ideas would
be most welcome.

Jason

You could install RTAI that operates between linux and the hardware.
It intercepts any interrupt usage by linux. It has a latency of 2-5us IIRC.

Do you use tasklets ?

Are you using PC hardware ? Why did you need to write you r own driver ?
The Kernel comes with an "extended" driver for the PC RTC (rtc.c) that
can be used to periodically wake up a program. You can enable the
extended RTC support as a module or a built-in driver with Kernel 2.4.
Of course if you want to periodically access some hardware it might be
useful to create your own driver using the RTC interrupt.

Did you do this in the driver or did you write a user land program ?
What do you mean by "prints out a message". Printing the message might
take much more than 122 sec and this might ask for confusion.

Maybe the interrupt works, but the userland program is not scheduled. In
my tests a year ago I did something similar using the standard extended
RTC driver. This driver counts the interrupts that it sees but the (high
"realtime" priority) userland program can't respond to as same is not
scheduled by the Kernel fast enough. I found that with 1 mSec interrupt
frequency my userland program can miss more than 100 interrupts, when
heavy IDE activity is done by a low priority user program. There are
"Kernel Preemption", "low latency", and "lock break" patches that help
reducing the probability of high latency situations, but even with
patches I saw such 100 msec latencies more than once an hour on a quite
fast PC. I did not even try to find out if a hardware interrupt itself
might be delayed or missed, which supposedly would only add to the
latency I found.

I don't know  "show_stack(0)" but maybe the situations where the macros
are included might be much to "dangerous" to call such a function. I
would rather just store the value in some global place and read it out
in some driver that you call from a userland program now and again.

Are there no macros explicitly disabling/enabling the interrupt ?

An occurring interrupt of course disables the interrupt in the processor
hardware, too, without using a macro.

A higher priority hardware interrupt might delay a lower level interrupt
without the processor noticing it at all.

The interrupts can be disabled individually by setting a mask in the
interrupt controller.

Generally {*filter*} Linux might be not suitable for what you want to do as
it does not guarantee any limit to interrupt or scheduling latency. I
did not use it but RTAI is the recommended way to handle that type of
problems.

-Michael

Yes, I'm using PC hardware, and yes, I saw the RTC application that
uses the RTC driver that comes with the kernel.  I wrote my own driver
because of the fact that user land is way too slow.  I want the stuff
I need to do when the timer goes off to run with minimal delay, in
kernel space, with interrupts shut off.  In other words, I need to
take over the system periodically.  ;-)

This was all done in my RTC driver.  There are no user-level apps
associated with what I'm doing.  "Prints out a message" means I call
'printk' when I detect a new max.  You make a good point about the
print taking too much time, but I don't believe that is happening in
this case.  It would be interesting to try your idea about just
storing the data and then polling it (perhaps using the /proc
interface) after I've run my test for awhile.  I will keep that in
mind for the future.

All very good points -- especially the last one.  Since my post, I've
been looking into and playing around with RTAI.  So far, the results
look good.  Using RTAI, I don't need to use the RTC as a periodic
timing source, I can just schedule a real time task to go off at
around the same rate.  Doing this lowered my jitter significantly --
to around 5-10 microseconds.  I am seeing occasional jitter of close
to a millisecond however (i.e. in a 20hr test, my realtime task has
been late by about a millisecond almost 300 times).  I'm not sure what
the reason for that is, but I suspect it's because I'm testing this on
a laptop with all kinds of "evil" processes and modules running.  When
I get back to work tomorrow I am going to run my test in the "real"
environment.  In this environment, only the absolute minimum modules
and processes will be running.  Hopefully this will result in better
numbers.

Thanks again for your input Michael.  All very good stuff!

Jason

RTAI claims that this should not happen. The RTAI timers _should_ offer
an accuracy of some sec. The linux running on top of RTAI should not
harm at all, and I suppose that you are not running any RTAI processes
that might keep the interrupt disabled.

Please keep us posted.

-Michael