Hi All,

Apologies in advance for my verbosity and if I have posted to
inappropriate groups.

I am encountering an issue which I wanted to pass by the gurus here.

I have a multi-threaded data aquisition application which, when
commanded, enters a mode of operation where it needs to sample/re-arm
an A/D card every 1 ms for a 60 second duration (i.e. 60000 times.)

It's a bare bones RedHat 9 2.4.20-8 Linux system, 512 MB, no swap,
with the only services running is syslogd and xinetd daemon services.
The xinetd daemon provides the occasional telnet, ftp.

I have max'ed the priorities of my application and it's threads. The
thread that reads the data out of the A/D card sleeps until there is
data available. All seems to run fine MOST of the time. I am able to
wake up at regular 1 ms intervals to read the data off the A/D card
and re-arm it for the next trigger.

However, every 24-25 seconds my app gets pre-empted for 2-15 ms
causing me to miss triggers. After looking into the problem and much
Googling/DejaNewsing/RTFMing, I believe the cause is due to the
kscand/Normal thread waking up and blocks my app from running.

I'm a bit puzzled by why this should happen. My system is configured
for no swap space and has enough physical RAM to handle any necessary
data buffers I need.

I would be truly grateful if the smarter-than-me Linux gurus out there
could shed some light on whether I can fix this problem.

What function does kscand/Normal provide even if I'm not using swap?
(My guess there is still some VM housekeeping thet the kernel needs to
manage, like cache, TLB's, etc., and there's no way to do without it
or disable it.)

Are there any kernel mods I could do to make things run better?

I have gotten my hands on an RTLinuxPro eval, but I'm unsure if it
will fix my problem. Will my A/D driver need tweaking/re-write to run
under RTLinux? Maybe RTAI is a better solution?

Are there other real time Linux extensions/patches that might be good
candidates for addressing this problem? If any one could provide me a
link to some more detailed discussions about Ingo Molnar's and Robert
Love's kernel scheduler patches, that would be appreciated too.

Thanks for your time.

Tom

From:	Michael Blanc
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Sun, 21 Nov 2004 09:27:59 -0800

Is there an ethernet controller installed on your system, (and enabled
during data acquisition)?

From:	Tom
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	29 Nov 2004 06:50:15 -0800

HI Michael,

Sorry for the response delay. Soon after my post, I was away on
holiday.

You raise a valid issue. Yes, the ethernet is enabled during
acquisition and sends occasional status messages back to a remote host
controlling the acqusition, however, from a lower priority thread.

This has raised my suspicions also. I am working on a method to
disable the status messages during acquisition to verfiy whether this
is a culprit.

Thanks for your post. Sorry for the delay in responding.

Tom

"Michael Blanc" wrote in message news:<1101057980.250649@news-1.nethere.net>...
> Is there an ethernet controller installed on your system, (and enabled
> during data acquisition)?

From:	Michael Blanc
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Mon, 29 Nov 2004 11:36:01 -0800

Hello Tom,

The FDDI controller may well be the key to the problem here. If there is
traffic on the line which generates an interrupt signal, then the service of
that interrupt might be sufficient to cause a small gap in the analog data
acquisition (though ~24 ms sounds excessive!). Anyways it would scarcely
matter at what priority a request gets scheduled which emmanates from the
transaction (of course if posting the request itself generates a pass
through the task scheduler, there goes some more of the 24 ms....).

A crude, though possibly effective method would be to disable the interrupt
on the FDDI before the 60 sec of data acq initiates, then re-enable it
after. Ofcourse, since you probably have "all the time in the world" before
onset, and after cessation of that 60 sec period, you could do something
more "formal" with LINUX.

Please let me know how this works out, either on this forum by the email.

Cordially,

M. Blanc

Tom wrote in message <1e7d6669.0411290650.378632f4@posting.google.com>...
>HI Michael,
>
>
>Sorry for the response delay. Soon after my post, I was away on
>holiday.
>
>You raise a valid issue. Yes, the ethernet is enabled during
>acquisition and sends occasional status messages back to a remote host
>controlling the acqusition, however, from a lower priority thread.
>
>This has raised my suspicions also. I am working on a method to
>disable the status messages during acquisition to verfiy whether this
>is a culprit.
>
>Thanks for your post. Sorry for the delay in responding.
>
>Tom
>
>
>
>
>"Michael Blanc" wrote in message
news:<1101057980.250649@news-1.nethere.net>...
>> Is there an ethernet controller installed on your system, (and enabled
>> during data acquisition)?

From:	Tom
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	30 Nov 2004 13:04:51 -0800

Hi Michael,

Yes, you have surmised my situation exactly.

Your approach might work, but how does one disable an interrupt from
user space? I have the impression it can only be done from
kernel/module space.

(I've written drivers before - my development schedule may be
*SLIGHTLY* affected with the impact of a new module development!)

Thanks,
Tom

"Michael Blanc" wrote in message news:<1101756862.926299@news-1.nethere.net>...
> Hello Tom,
>
> The FDDI controller may well be the key to the problem here. If there is
> traffic on the line which generates an interrupt signal, then the service of
> that interrupt might be sufficient to cause a small gap in the analog data
> acquisition (though ~24 ms sounds excessive!). Anyways it would scarcely
> matter at what priority a request gets scheduled which emmanates from the
> transaction (of course if posting the request itself generates a pass
> through the task scheduler, there goes some more of the 24 ms....).
>
> A crude, though possibly effective method would be to disable the interrupt
> on the FDDI before the 60 sec of data acq initiates, then re-enable it
> after. Ofcourse, since you probably have "all the time in the world" before
> onset, and after cessation of that 60 sec period, you could do something
> more "formal" with LINUX.
>
> Please let me know how this works out, either on this forum by the email.
>
> Cordially,
>
> M. Blanc
>
>
> Tom wrote in message <1e7d6669.0411290650.378632f4@posting.google.com>...
> >HI Michael,
> >
> >
> >Sorry for the response delay. Soon after my post, I was away on
> >holiday.
> >
> >You raise a valid issue. Yes, the ethernet is enabled during
> >acquisition and sends occasional status messages back to a remote host
> >controlling the acqusition, however, from a lower priority thread.
> >
> >This has raised my suspicions also. I am working on a method to
> >disable the status messages during acquisition to verfiy whether this
> >is a culprit.
> >
> >Thanks for your post. Sorry for the delay in responding.
> >
> >Tom
> >
> >
> >
> >
> >"Michael Blanc" wrote in message
> news:<1101057980.250649@news-1.nethere.net>...
> >> Is there an ethernet controller installed on your system, (and enabled
> >> during data acquisition)?

From:	Michael Blanc
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Tue, 30 Nov 2004 16:56:12 -0800

Tom,
After I sent off that suggestion, I had some other thoughts. To wit: there
should be other was to disable ethernet transactions without low-level
messing around - but you would almost certainly have to have root
priviledge. Options might be:

1> Reconfigure the FDDI controller to ignore all incoming packets -
essentially to go into a state of absolute quiescience during the 60 second
period - and then restore normal functioning after. Or

2> Uninstall the driver which handles the controller, then re-install. This
is klunky, but should work, hopefully without a reboot.

But I think that we agree in general. The approach is not to move around
some basket which catches the falling fruit, but to uproot the tree
altogether (and replant in 60 sec).

I am curious - are there other users on this one linux machine besides
yourself and the data acq task? Are the A/D samples obtained through
interrupt, or polling?

Regards,

Mike

From:	Tom
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	1 Dec 2004 08:09:46 -0800

Hi Michael,

"Michael Blanc" wrote in message news:<1101862469.989183@news-1.nethere.net>...
> Tom,
> After I sent off that suggestion, I had some other thoughts. To wit: there
> should be other was to disable ethernet transactions without low-level
> messing around - but you would almost certainly have to have root
> priviledge. Options might be:
>
> 1> Reconfigure the FDDI controller to ignore all incoming packets -
> essentially to go into a state of absolute quiescience during the 60 second
> period - and then restore normal functioning after. Or
>
> 2> Uninstall the driver which handles the controller, then re-install. This
> is klunky, but should work, hopefully without a reboot.

Good suggestions, thank you. I will continue to try out things.
>
> But I think that we agree in general. The approach is not to move around
> some basket which catches the falling fruit, but to uproot the tree
> altogether (and replant in 60 sec).
>
> I am curious - are there other users on this one linux machine besides
> yourself and the data acq task? Are the A/D samples obtained through
> interrupt, or polling?

No, there are no other users. The A/D app runs as root renice'd to
-20. The A/D card is armed for trigger then the app sleeps until the
sample set is collected. So it is interrupt driven, however, I have
tried polling, but without any difference in results. The whole
process is repeated for the 60 seconds. Here's some psuedo code to
illustrate:

struct sampleSet {
struct timeval t[60000];
short buff[60000*256];
struct oh overhead[60000]; // various overhead data required by
A/D card.
} sampSet;

struct sampSet* ss = &sampSet;

// the overhead is large. The total size allocated is around 128 MB

// for 60000 triggers @ 1KHz ==> i.e. 1 min
for( i = 0, index = 0; i < 60000; i++, index += 256 ){

// timestamp this buffer
gettimeofday(&ss->t[i]);

// arm the A/D immediate return
acquire();

// sleep til sample set collected sample set size is 256 samples
at 250 MS/s
// This sleeps for approx. 1.024 usec
waitForEndOfAcquisition();

// DMA transfer out of A/D card takes approx. 40 usec
readData(&ss->buff[index], &ss->overhead[i]);

}

// write timestamps and buffer to disk
for( i = 0, index = 0; i < 60000; i++, index += 256 ){
fwrite( &ss->t[i], ... );
fwrite( &ss->buff[index], sizeof(short),256, ... );
fwrite( &ss->overhead[i], ... );
}

As you can see, and as "top" confirms, CPU utilization is low, around
2-5%. I have also confirmed (via scope), the trigger source is
constant.

Every 23-25 seconds, kscand/Normal daemon wakes up and takes 5-15 ms
to run. Causing my delta's (ie t[i+1] - t[i]) to exceed 1 ms. I'm
currently flailing with sysctl and trying to tame/understand the
kscand/Normal behavior. I'm also trying to determine if kscand/Normal
is the problem or an interrupt state (such as ethernet that happens to
occur during kscand/Normal) is causing my headaches.

I found the following post very interesting and similar to my
situation:
http://groups.google.com/groups?q=kscand/Normal&hl=en&lr=&safe=off&selm=fa.mrdk84l.bgkkq3%40ifi.uio.no&rnum=1
I have contacted the author and, hopefully, he can provide some
additonal insight into this problem/situation.

I'll keep posting as long as anybody is listening...

Thanks for your reply,
Tom

From:	Michael Blanc
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Wed, 1 Dec 2004 12:48:35 -0800

Tom,
I assume that your sample clock is 250K not 250M, and that the inter-burst
(trigger) period is ~1 ms, not 1 us. If that is the case, your setup seems
perfectly reasonable.

Since you have identified the interfering task, it seems like some solution
shoud be available.

Try this test: just unplug the ethernet cable and then take some data! If
incoming packets are what initiate the scheduling of the blocking task,
you'll know. (But then, you probably tried _that one_ already...)

Regards,

Mike

From:	Tom
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	3 Dec 2004 08:21:14 -0800

Hi MIchael,

Sorry for the delay. Yesterday spent debugging...

"Michael Blanc" wrote in message news:<1101934013.441370@news-1.nethere.net>...
> Tom,
> I assume that your sample clock is 250K not 250M, and that the inter-burst
> (trigger) period is ~1 ms, not 1 us. If that is the case, your setup seems
> perfectly reasonable.

The sample rate is indeed 250 MS/s and the trigger period is 1 ms ( 1
KHz ). So one 256 sample set takes 256/250e6 = 1.024e-6 secs --> 1.024
microsecs. The advertised (and measured) transfer rate out the card is
40-50 microseconds. So, *in theory*, I should have plenty of time to
get ready and sleep for the next trigger.

>
> Since you have identified the interfering task, it seems like some solution
> shoud be available.

Fingers x'ed. I am digging around in the kernel source to try and
understand why the kscand/Normal task is taking 10-15 ms to run when
no swap is enabled. The adventure continues... Still have some sysctl
options I need to explore.

From /usr/src/linux-2.4/Documentation/sysctl/vm.txt:

kswapd:

Kswapd is the kernel swapout daemon. That is, kswapd is that
piece of the kernel that frees memory when it gets fragmented
or full. Since every system is different, you'll probably want
some control over this piece of the system.

The numbers in this page correspond to the numbers in the
struct pager_daemon {tries_base, tries_min, swap_cluster
}; The tries_base and swap_cluster probably have the
largest influence on system performance.

tries_base The maximum number of pages kswapd tries to
free in one round is calculated from this
number. Usually this number will be divided
by 4 or 8 (see mm/vmscan.c), so it isn't as
big as it looks.
When you need to increase the bandwidth to/from
swap, you'll want to increase this number.
tries_min This is the minimum number of times kswapd
tries to free a page each time it is called.
Basically it's just there to make sure that
kswapd frees some pages even when it's being
called with minimum priority.
swap_cluster This is the number of pages kswapd writes in
one turn. You want this large so that kswapd
does it's I/O in large chunks and the disk
doesn't have to seek often, but you don't want
it to be too large since that would flood the
request queue.

Hmmm... maybe some interesting stuff here to play with.

>
> Try this test: just unplug the ethernet cable and then take some data! If
> incoming packets are what initiate the scheduling of the blocking task,
> you'll know. (But then, you probably tried _that one_ already...)

Yes, good point. Did this yesterday. No difference in results. In
fact, nothing but the kernel and the app running. Ran
"/etc/rc.d/init.d/network stop". Looked into disabling the on-board
ethernet via BIOS, but not an option. So the best I can do is disable
network service and unplug cable. SOSDD...

>
> Regards,
>
> Mike

Thanks for the continued interest and help.

Cheers,
Tom

From:	Michael Blanc
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Sun, 5 Dec 2004 09:18:30 -0800

Tom wrote in message <1e7d6669.0412030821.1ee31f81@posting.google.com>...
>Hi MIchael,

>The sample rate is indeed 250 MS/s and the trigger period is 1 ms ( 1
>KHz ). So one 256 sample set takes 256/250e6 = 1.024e-6 secs --> 1.024
>microsecs. The advertised (and measured) transfer rate out the card is
>40-50 microseconds. So, *in theory*, I should have plenty of time to
>get ready and sleep for the next trigger.
>

This is one interesting sampler! Can it possibly be any wider than 8 bits?
If a commercial one, which one is it - I am curious.

Anyways, I guess that we have ruled out the possibility that the ethernet
has anything to do with launching the offending daemon. (And I did follow
the link you provided to the chap in Houston who seemed to have the
identical problem.)

The kernel settings which you quoted don't look like they would be
sufficient to keep the daemon from being a nuissance. On the other hand, its
functions certainly seem like they could be suspended for brief intervals,
such as 60 seconds, without endangering the systems well-being. (Especially
since it only seems to launch every 25 seconds or so - that would only
prevent it once or twice.)

My solution would be to customize the kernel code with a gate flag which
would block the daemon from being launched. Such a flag could be set from
the root (via a new system call) just before the onset of the 60 second
acquisition interval, and then reset afterwards. In that way, the daemon
would only sporadically get blocked once or twice, but would otherwise
continue to function normally.

If the scheduler code won't afford a disabling flag, perhaps it the logic
could be put inside the daemon code itself - an instant
return-before-deployment for example. A brief consultation with a linux
expert could clear this up. If any have followed this thread thus far, maybe
one would comment. Otherwise smoe more usenet searching, and possibly a
posting to another group, as the question becomes a more generic one, not
specifically depending on realtime issues.

Good luck!

MB

From:	Mark H Johnson
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Wed, 01 Dec 2004 11:56:48 -0600

Tom wrote:
[snip]
> No, there are no other users. The A/D app runs as root renice'd to
> -20. The A/D card is armed for trigger then the app sleeps until the
> sample set is collected. So it is interrupt driven, however, I have
> tried polling, but without any difference in results. The whole
> process is repeated for the 60 seconds. Here's some psuedo code to
> illustrate:
>

renice will not get you a real time priority. On Linux, you should be
calling sched_setscheduler doing something like this...

#include
....

struct sched_param schp;

memset(&schp, 0, sizeof(schp));
schp.sched_priority = 30; // RT priority, max is 99 on Linux

if (sched_setscheduler(0, SCHED_FIFO, &schp) != 0) {
perror("sched_setscheduler");
return -1;
}

This will set your scheduling type to real time FIFO - first in, first
out. At this point, you basically run above everything that is not RT
and above all RT tasks below your priority. Interrupts and higher
priority RT tasks can cause delays but that is about it.

If your system has a copy of chrt, you can use that as well to set RT
priorities.
http://tech9.net/rml/schedutils/

I posted a similar comment about this on 11/22 on comp.realtime, perhaps
you missed it. When you cross post like this, you really should pick
just one newsgroup for follow ups.

--Mark

From:	Pat Ford
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Wed, 1 Dec 2004 11:48:26 -0500

"Tom" wrote in message
news:1e7d6669.0412010809.5a4654bc@posting.google.com...
> Hi Michael,
>
> "Michael Blanc" wrote in message
news:<1101862469.989183@news-1.nethere.net>...
> I found the following post very interesting and similar to my
> situation:
>
http://groups.google.com/groups?q=kscand/Normal&hl=en&lr=&safe=off&selm=fa.mrdk84l.bgkkq3%40ifi.uio.no&rnum=1
> I have contacted the author and, hopefully, he can provide some
> additonal insight into this problem/situation.
>
> I'll keep posting as long as anybody is listening...
>
> Thanks for your reply,
> Tom

I'm watching and learning!
Pat

From:	Robert Redelmeier
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Tue, 30 Nov 2004 22:24:42 GMT

In comp.os.linux.development.system Tom wrote:
> Your approach might work, but how does one disable an
> interrupt from user space? I have the impression it can
> only be done from kernel/module space.

Once upon a time, PCs contained 8259 PICs that had a writable
IRQ mask at some I/O address. They still have the virtual
equivalent semi-retired in the Southbridge. ioperm() will let
you write this mask, _if you dare_!

More and more often now, people use the APIC which I believe
has a similar mask living at some even more hidden I/O addr.

-- Robert

From:	Jan Knutar
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Sat, 27 Nov 2004 14:24:49 +0200

Tom wrote:

> Are there other real time Linux extensions/patches that might be good
> candidates for addressing this problem?

Ingo Molnar has been making realtime patches lately, which are supposedly
pretty close to achieving hard realtime capability.

http://people.redhat.com/mingo/realtime-preempt/

From:	Joseph Gwinn
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Sun, 28 Nov 2004 19:07:45 GMT

In article ,
Jan Knutar wrote:

> Tom wrote:
>
> > Are there other real time Linux extensions/patches that might be good
> > candidates for addressing this problem?
>
> Ingo Molnar has been making realtime patches lately, which are supposedly
> pretty close to achieving hard realtime capability.
>
> http://people.redhat.com/mingo/realtime-preempt/

I've been following this too. They are still working out numerous
stability issues, and a bunch of timing glitches, but the current test
builds already achieve respectable soft realtime behavior, with only two
in a million samples exceeding 50 microseconds on a 600 MHz processor.
However, some of those "two in a million" samples exceed sixteen
milliseconds, so there is some work to be done. The current suspicion
is that some code paths don't enable the interrupts when they should
[Lee Revell, Mon Nov 01 2004 - 13:00:40 EST].

I would be surprised if it took more than a year or perhaps two for the
Linux community to find and fix the various causes, and it seems likely
that Linux itself will soon become realtime enough for many traditional
embedded realtime applications. This has to raise the pressure on both
the traditional RTOS vendors and on the various realtime linux vendors,
as well as on Microsoft and Sun.

Joe Gwinn

From:	Bob Hauck
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Sun, 28 Nov 2004 21:00:33 -0500

["Followup-To:" header set to comp.os.linux.development.system.]
On Sun, 28 Nov 2004 19:07:45 GMT, Joseph Gwinn wrote:
> In article ,
> Jan Knutar wrote:
>
>> Tom wrote:
>>
>> > Are there other real time Linux extensions/patches that might be good
>> > candidates for addressing this problem?

Note that there are some motherboards out there that just can't be used
for real-time (most laptops for instance). Before you do anything else
you should get the RTAI live CD:

This will run some tests on your motherboard and tell you what the
expected latency and timing jitter are under RTAI. RTLinux ought to
give roughly similar results.

>> Ingo Molnar has been making realtime patches lately, which are
>> supposedly pretty close to achieving hard realtime capability.
>>
>> http://people.redhat.com/mingo/realtime-preempt/
>
> I've been following this too. They are still working out numerous
> stability issues, and a bunch of timing glitches, but the current test
> builds already achieve respectable soft realtime behavior, with only
> two in a million samples exceeding 50 microseconds on a 600 MHz
> processor.

One problem with this approach is that it is statistical. A million
cycles at one kilohertz is only a little over 15 minutes. For some
applications it is ok to glitch every 15 minutes or so, for others not.

Another problem is that interrupt disables and such are determined not
just by the real-time code path, but also by every other kernel module
or driver that's loaded at the same time. If you want to use such a
kernel in a hard real-time application you have to validate it with the
combination of modules that you are using. In addition, driver writers
aren't in general going to worry too much about interrupt disable times,
because few users care about that, so it becomes an ongoing problem to
re-validate every new version.

A sub-kernel approach like RTAI or RTLinux can guarantee timing
regardless of the drivers Linux uses. Well, almost, there are still
some things to consider (e.g. PCI bus hogging, SMI), but the universe of
possible problems is a lot smaller. You mostly only need to worry about
the specific drivers you will be using on the real-time side, not every
driver that's loaded.

There are disadvantages as well of course, because you essentially have
to partition your application, but if you have a hard real-time
requirement it is usually going to be easier to meet it with the RTLinux
approach than with a general-purpose kernel.

> I would be surprised if it took more than a year or perhaps two for
> the Linux community to find and fix the various causes, and it seems
> likely that Linux itself will soon become realtime enough for many
> traditional embedded realtime applications.

It is realtime enough for many applications now, but not so much for
others. If an application can tolerate an occasional missed deadline,
or don't have any strict timing requirements at all, then you can use
the preemptible kernel approach. Lots of embedded apps are like that,
more than you might think.

But there is another reason for reducing latency in the kernel even on
the desktop, and that is to support multimedia apps better. I think
Linux is just about there in that respect.

--
-| Bob Hauck
-| To Whom You Are Speaking
-| http://www.haucks.org/

From:	Tom
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	29 Nov 2004 06:53:20 -0800

HI Bob,

Thanks. I will try this.

Tom

Bob Hauck wrote in message news:...

>
> Note that there are some motherboards out there that just can't be used
> for real-time (most laptops for instance). Before you do anything else
> you should get the RTAI live CD:
>
>
>
> This will run some tests on your motherboard and tell you what the
> expected latency and timing jitter are under RTAI. RTLinux ought to
> give roughly similar results.

From:	patrick carosso
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Sat, 20 Nov 2004 18:38:30 GMT

>
> I have gotten my hands on an RTLinuxPro eval, but I'm unsure if it
> will fix my problem. Will my A/D driver need tweaking/re-write to run
> under RTLinux? Maybe RTAI is a better solution?
>
> Are there other real time Linux extensions/patches that might be good
> candidates for addressing this problem? If any one could provide me a
> link to some more detailed discussions about Ingo Molnar's and Robert
> Love's vkernel scheduler patches, that would be appreciated too.
>

I've some experience about real-time issues on Linux and RTAI-linux.
Maybe I can help you.
But, first of all:
what's the exact code you use for starting the threads?
Patrick

From:	Tom
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	29 Nov 2004 06:42:13 -0800

Hi Patrick,

My sincerest apologies. Soon after my post, I was away on holiday.
Thank you for your reply.

In response to your post, here is my code for starting the "acquire
thread":

// Init the acquire thread attributes
if( pthread_attr_init(&tattr) ){
syslog(LOG_ERR,"\nERROR: Could not initialize thread
attributes.");
return -4;
}

// thread has FIFO scheduling policy
if( pthread_attr_setschedpolicy(&tattr, SCHED_FIFO) ){
syslog(LOG_ERR,"\nERROR: Could not set thread scheduling
policy.");
return -5;
}

// created/child threads have explict rather than inherited
// scheduling attributes
if( pthread_attr_setinheritsched( &tattr, PTHREAD_EXPLICIT_SCHED) ){
syslog(LOG_ERR,"\nERROR: Could not set thread inheritance
policy.");
return -6;
}

// the acquire thread has max priority
tparam.sched_priority = sched_get_priority_max(SCHED_FIFO);
if( tparam.sched_priority == -1 ){
syslog(LOG_ERR,"\nERROR: Could not get thread priority.");
return -7;
}

// set acquire thread priority
if( pthread_attr_setschedparam(&tattr, &tparam) ){
syslog(LOG_ERR,"\nERROR: Could not set thread priority.");
return -8;
}

// create/spawn acquire thread
if( pthread_create( &acquireTid, &tattr, (void*)startAcquire, NULL )
){
syslog(LOG_ERR,"ERROR: Could not create acquire thread.\n");
return -9;
}

// detach thread
pthread_detach(acquireTid);

Thanks. Sorry for the delay.

Tom

"patrick carosso" wrote in message news:...
> >
> > I have gotten my hands on an RTLinuxPro eval, but I'm unsure if it
> > will fix my problem. Will my A/D driver need tweaking/re-write to run
> > under RTLinux? Maybe RTAI is a better solution?
> >
> > Are there other real time Linux extensions/patches that might be good
> > candidates for addressing this problem? If any one could provide me a
> > link to some more detailed discussions about Ingo Molnar's and Robert
> > Love's vkernel scheduler patches, that would be appreciated too.
> >
>
> I've some experience about real-time issues on Linux and RTAI-linux.
> Maybe I can help you.
> But, first of all:
> what's the exact code you use for starting the threads?
> Patrick

From:	patrick carosso
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Tue, 30 Nov 2004 09:54:10 GMT

What you're doing seems OK, as far as I know, except
that you're not locking the image of the process:
see mlock/mlockall.
I'm not sure if it will help, but you can try.

Before using RTAI, I also made some experiments
with SCHED_FIFO tasks. They worked rather good,
in average, but there were times they had a delay
of many ms. And I was working at 100 Hz only...
It seemed to me that now and then the kernel activity
blocked the tasks. Moreover, I heard from an industrial machines
expert that on standard linux it's not possible to have periodic
tasks over 100Hz with acceptable jitter.

So I've have installed RTAI Linux 3.0r4
on a 2.4.25 kernel (RedHat 9) on an indutrial 1.3Ghz PC and
made some test with oscilloscope: under 100% load
the real time 100Hz task had some microseconds jitter only.
If you're interested, I can give you some hints on how to
patch, configure and compile the kernel.
It tooks me some time... but it was the first time
I was doing such things. You seem to be more expert than me...
Please let me know the progress of your work : I'm very interested.

Patrick

From:	Tom
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	30 Nov 2004 12:44:25 -0800

Hi Patrick,

"patrick carosso" wrote in message news:...
> What you're doing seems OK, as far as I know, except
> that you're not locking the image of the process:
> see mlock/mlockall.
> I'm not sure if it will help, but you can try.

Good point. Yes, I do a mlockall(MCL_CURRENT|MCL_FUTURE) at startup.
Doesn't seem to have any effect on my problem.

> Before using RTAI, I also made some experiments
> with SCHED_FIFO tasks. They worked rather good,
> in average, but there were times they had a delay
> of many ms. And I was working at 100 Hz only...
> It seemed to me that now and then the kernel activity
> blocked the tasks. Moreover, I heard from an industrial machines
> expert that on standard linux it's not possible to have periodic
> tasks over 100Hz with acceptable jitter.

Hmmm, as I am beginning to discover... Do you know anything about the
CONFIG_LOW_LATENCY option on kernel re-builds?

>
> So I've have installed RTAI Linux 3.0r4
> on a 2.4.25 kernel (RedHat 9) on an indutrial 1.3Ghz PC and
> made some test with oscilloscope: under 100% load
> the real time 100Hz task had some microseconds jitter only.
> If you're interested, I can give you some hints on how to
> patch, configure and compile the kernel.

Absolutely. Any hints would be appreciated, since I was under the
impression that RTAI used kernel trees from kernel.org and not
distributions. Not that it should matter but I would assume there a
few path changes/mods that need to be passed to the RTAI
Makefile/install scripts. Thanks!

Still hammering away,
Tom

From:	7
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Fri, 19 Nov 2004 23:18:17 GMT

Tom wrote:

I would say you are on a losing streak - PCs are not
generally beasts of burden for this kind of work.
You want an embedded controller with a big buffer
and some nice C program to do the donkey work
and pass it up a serial connection to the PC.

From:	Stefan M. Petters
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Mon, 22 Nov 2004 10:54:09 +1100

Hi,

7 wrote:
> I would say you are on a losing streak - PCs are not
> generally beasts of burden for this kind of work.
> You want an embedded controller with a big buffer
> and some nice C program to do the donkey work
> and pass it up a serial connection to the PC.
While I think it is possible to do this with a PC, I would argue it is
much cheaper (in terms of time and money) to have some embedded
controller do that kind of stuff for you. It doesn't really matter what
kind of processor you use, so just go for a reasonable cheap board (it's
really not a lot you want to do and you don't want to solder too much on
your own), which has the bit's you want to have included (memory,
clocking, serial interface and a bus for the ADC).

Regards,
Stefan.

From:	Mark H Johnson
Subject:	Re: Real-time pre-emption kswapd/kscand questions
Date:	Mon, 22 Nov 2004 13:43:06 -0600

Tom wrote:
> I have a multi-threaded data aquisition application which, when
> commanded, enters a mode of operation where it needs to sample/re-arm
> an A/D card every 1 ms for a 60 second duration (i.e. 60000 times.)
>
> It's a bare bones RedHat 9 2.4.20-8 Linux system, 512 MB, no swap,
> with the only services running is syslogd and xinetd daemon services.
> The xinetd daemon provides the occasional telnet, ftp.
>
Red Hat does a reasonable job of building kernels with low latency
patches, but I find I need preemption patches as well to get a good
result. For a 2.4.20 kernel, start with the stock kernel from kernel.org
and then apply the following patches
preempt-kernel-rml-2.4.20-1.patch
preempt-log-rml-2.4.18-1.patch [optional]
2.4.20-low-latency.patch
proc-add-rt-info-rml-2.4.18-1.patch [optional]
cpu-affinity-syscall-rml-2.4.20-1.patch [optional, if you use SMP]
If you don't know where to get these, send a message separately and I
can send you locations for all of them.

The alternative is to use a RT extension such as RTLinux or RTAI. (as
already noted by another person)

> I have max'ed the priorities of my application and it's threads. The
> thread that reads the data out of the A/D card sleeps until there is
> data available. All seems to run fine MOST of the time. I am able to
> wake up at regular 1 ms intervals to read the data off the A/D card
> and re-arm it for the next trigger.
>
One msec is a little fast for a stock Linux but may be OK depending on
how you set the system up. A few questions...
- I know you don't have swap, but do you use the following
mlockall(MCL_CURRENT|MCL_FUTURE)
It is possible that the whole application is not in memory (e.g., read
only sections paged from the disk).
- Are you using sched_setpriority (or an equivalent method) to set RT
FIFO priorities?

> However, every 24-25 seconds my app gets pre-empted for 2-15 ms
> causing me to miss triggers. After looking into the problem and much
> Googling/DejaNewsing/RTFMing, I believe the cause is due to the
> kscand/Normal thread waking up and blocks my app from running.
>
Doesn't sound like you have all the priorities set right. The other
possibility could be a hardware mediation problem. I've found that using
something like
hdparm -X66 /dev/hda
is necessary to use UDMA2 (instead of UDMA4 or 5). Unless I do this, I
get latencies of > 1 msec even on a preempt / low latency kernel when
the kernel has to access RAM.

> [snip]
>
> Are there other real time Linux extensions/patches that might be good
> candidates for addressing this problem? If any one could provide me a
> link to some more detailed discussions about Ingo Molnar's and Robert
> Love's kernel scheduler patches, that would be appreciated too.
>
The linux kernel mailing list has a big bunch of messages in them. You
could go to groups.google.com and search for linux.kernel. The more
recent ones are all 2.6 related but the historical messages have the
results I've seen with 2.4 kernels (which is what I use in a big RT
application).

Good luck.
--Mark