Topics

Keyboard latency with multiple JTAlert processes running.


Patrick Skerrett
 

Hey folks,

Sorry I'm sure this is an edge case, but I am a Flexradio user and I typically operate WSJT-X with at least 4 bands running simultaneously. I will feed each slice of the Flex to a different WSJT-X process, and then I will run an equal number of JTAlert instances, each capturing the UDP stream of the individual WSJT-X process.

I have noticed that as I add more JTAlert instances, my Keyboard latency will decrease linearly. By the time I have reached 4 processes, the latency is to the point where typing is very difficult. 

My system is not taxed at all with all these programs running, I have a 12-core 4.0Ghz processor, and 32GB ram with a very fast GPU, and NVME storage. CPU usage while decoding 4 bands is 10- 20% & memory consumption is about 50%. 

I have tried debugging for most of the day, and can confirm that the latency only happens with multiple JTalert instances running. I do not feel any keyboard sluggishness when I add more WSJT-x or Flexradio slices. 


Has anyone encountered this before? 

I am running version 2.16.13 of jtalert, 2.2.2 of WSJT-x and 15.8.0 of Dxkeeper. 

Attached are some screenshots of my setup + a view of my system utilization when this is all running.

Thanks. 



W9PDS






Patrick Skerrett
 

I should add that I tested both with a unique UDP port per WSJT-X bound to 127.0.0.1 per instance, and also attempted with all WSJT-X instances configured for multicast using the same port. Both configurations were functional, but both had the extreme latency after about 4 running instances. 

W9PDS


m_scot_alex
 

I mentioned this a couple of times over the last couple of years. I can't say exactly when it started, but there was a definite point at which it started and I suspect it's a keyboard trapping issue.

73,

Mike - N8MSA


Patrick Skerrett
 

Thanks for confirming Mike. Does anyone know the process for filing a bug with the devs here?

73,
W9PDS


Michael Black
 


Don't think a bug report will help....this sounds like a system problem even though JTAlert may be the canary and there's unlikely to be anything in the debug log to see what's going on.

Mike W9MDB





On Wednesday, September 23, 2020, 08:30:52 AM CDT, Patrick Skerrett <patrick@...> wrote:


Thanks for confirming Mike. Does anyone know the process for filing a bug with the devs here?

73,
W9PDS


Patrick Skerrett
 

Hi Mark, I have been through those steps yesterday and spent a significant amount of time ensuring that the correlation between keyboard latency & JTAlert is not just a 'canary'. I have also created a video show how the system behaves as jtalert is scaled up. Notice the only thing changing here is additional instances of jtalert being brought up, but my typing becomes next to impossible even with a 15% cpu load:

 

 

https://youtu.be/G97HrJ9k7sE


Michael Black
 

Have you tried uninstalling your keyboard driver?

Mike W9MDB




On Wednesday, September 23, 2020, 10:26:23 AM CDT, Patrick Skerrett <patrick@...> wrote:


Hi Mark, I have been through those steps yesterday and spent a significant amount of time ensuring that the correlation between keyboard latency & JTAlert is not just a 'canary'. I have also created a video show how the system behaves as jtalert is scaled up. Notice the only thing changing here is additional instances of jtalert being brought up, but my typing becomes next to impossible even with a 15% cpu load:

 

 

https://youtu.be/G97HrJ9k7sE


Patrick Skerrett
 

Yes we did & also ran through the DISM. Just for completeness we also changed keyboards (ensured we are not using anything wireless, etc). 

I'm open to try anything, but feel free to read through my post & view the video. These fixes proposed are for general USB latency or latency only under high system load. In this use-case its specifically deteriorates with more instances of jtalert running. 



Michael Black
 

Next thing is to try DriverEasy and see if you don't have some drivers to update.

Mike W9MDB




On Wednesday, September 23, 2020, 10:56:39 AM CDT, Patrick Skerrett <patrick@...> wrote:


Yes we did & also ran through the DISM. Just for completeness we also changed keyboards (ensured we are not using anything wireless, etc). 

I'm open to try anything, but feel free to read through my post & view the video. These fixes proposed are for general USB latency or latency only under high system load. In this use-case its specifically deteriorates with more instances of jtalert running. 



Patrick Skerrett
 

I should note that I am current on .NET 4.8 with the  2020-KB4576484 update. 

 


HamApps Support (VK3AMA)
 

On 23/09/2020 6:54 am:

Hey folks,

Sorry I'm sure this is an edge case, but I am a Flexradio user and I typically operate WSJT-X with at least 4 bands running simultaneously. I will feed each slice of the Flex to a different WSJT-X process, and then I will run an equal number of JTAlert instances, each capturing the UDP stream of the individual WSJT-X process.

I have noticed that as I add more JTAlert instances, my Keyboard latency will decrease linearly. By the time I have reached 4 processes, the latency is to the point where typing is very difficult.

My system is not taxed at all with all these programs running, I have a 12-core 4.0Ghz processor, and 32GB ram with a very fast GPU, and NVME storage. CPU usage while decoding 4 bands is 10- 20% & memory consumption is about 50%.

I have tried debugging for most of the day, and can confirm that the latency only happens with multiple JTalert instances running. I do not feel any keyboard sluggishness when I add more WSJT-x or Flexradio slices.


Has anyone encountered this before?

I am running version 2.16.13 of jtalert, 2.2.2 of WSJT-x and 15.8.0 of Dxkeeper.

Attached are some screenshots of my setup + a view of my system utilization when this is all running.

Thanks.

W9PDS
OM,

Try temporarily turning off monitor on all your WSJTX instances so there is no new decodes processing in WSJTX and by extension no new decodes processing in JTAlert. Do you still get the keyboard latency?

de Laurie VK3AMA


Bill W2PKY
 

Flex 6700 i7 4790K 4Ghz PC running multiple instances of WSJT-X and JTAlert just like Pat. Notice keyboard clicks are delayed by JTAlert also. Tried turning off Monitor but did not restore keyboard speed. 
Bill W2PKY


HamApps Support (VK3AMA)
 

On 25/09/2020 10:42 am, Bill W2PKY wrote:
Flex 6700 i7 4790K 4Ghz PC running multiple instances of WSJT-X and JTAlert just like Pat. Notice keyboard clicks are delayed by JTAlert also. Tried turning off Monitor but did not restore keyboard speed. 
Bill W2PKY

Bill,

How many instances of JTAlert/WSJTX?

When you turned off Monitor, was that for all WSJT-X instances (it needed to be) and did you wait until WSJTX stopped producing decodes?

In the past there have been issues that were tracked down to a specific version of SmartSDR. What version are you running? Is it a recent upgrade? If this is a recent upgrade was this JTAlert issue apparent with the older SmartSDR version?

de Laurie VK3AMA


HamApps Support (VK3AMA)
 

On 25/09/2020 10:42 am, Bill W2PKY wrote:
Flex 6700 i7 4790K 4Ghz PC running multiple instances of WSJT-X and JTAlert just like Pat. Notice keyboard clicks are delayed by JTAlert also. Tried turning off Monitor but did not restore keyboard speed. 
Bill W2PKY

BIll,

Please use the "Help" -> "Contact Support" menu, on the main JTAlert window, to send me your JTAlert files for analysis.

de Laurie VK3AMA



HamApps Support (VK3AMA)
 

On 23/09/2020 6:54 am, Patrick Skerrett wrote:

Has anyone encountered this before? 

I am running version 2.16.13 of jtalert, 2.2.2 of WSJT-x and 15.8.0 of Dxkeeper. 

W9PDS


Has this always been a problem with your Flex and multi-instance JTAlert or is it a recent manifestation? If it is recent, what was the last version of JTAlert that didn't exhibit this behavior?

de Laurie VK3AMA



Bill Barrett
 

Hello Laurie-

I have noticed this for a long time. Never knew what caused it. Always
assumed it was due to running multiple instances of WSJT-X.
The reason Flex users notice it is because we have multiple slices and
run multiple instances of WSJT-X.
The keyboard delay is definitely related to using JTAlert. Without
JTAlert the keyboard response if fast.
In my case with the 6700 can run as many as 8 instances of "X" and
JTAlert. Then the keyboard response seems stacked up.
When pressing down on the left or right arrow key may take a second
before there is a response on the screen and then the cursor moves
several positions.
Hard to tell where the interference is happening, however, seems like
the keyboard interrupt handler is stalled.
Hope this helps.
Bill W2PKY


On Thu, Sep 24, 2020 at 10:44 PM HamApps Support (VK3AMA)
<vk3ama.ham.apps@...> wrote:

On 23/09/2020 6:54 am, Patrick Skerrett wrote:

Has anyone encountered this before?

I am running version 2.16.13 of jtalert, 2.2.2 of WSJT-x and 15.8.0 of Dxkeeper.

W9PDS


Has this always been a problem with your Flex and multi-instance JTAlert or is it a recent manifestation? If it is recent, what was the last version of JTAlert that didn't exhibit this behavior?

de Laurie VK3AMA




--
Bill Barrett
352-437-4758


Michael Black
 

Have you looked at the "Ease of Access keyboard settings"?
There's lots of those things and toggling them off one at a time might find a culprit.

Mike W9MDB






On Friday, September 25, 2020, 06:42:27 AM CDT, Bill Barrett <billbarrett174@...> wrote:


Hello Laurie-

I have noticed this for a long time. Never knew what caused it. Always
assumed it was due to running multiple instances of WSJT-X.
The reason Flex users notice it is because we have multiple slices and
run multiple instances of WSJT-X.
The keyboard delay is definitely related to using JTAlert. Without
JTAlert the keyboard response if fast.
In my case with the 6700 can run as many as 8 instances of "X" and
JTAlert. Then the keyboard response seems stacked up.
When pressing down on the left or right arrow key may take a second
before there is a response on the screen and then the cursor moves
several positions.
Hard to tell where the interference is happening, however, seems like
the keyboard interrupt handler is stalled.
Hope this helps.
Bill W2PKY


On Thu, Sep 24, 2020 at 10:44 PM HamApps Support (VK3AMA)
<vk3ama.ham.apps@...> wrote:
>
> On 23/09/2020 6:54 am, Patrick Skerrett wrote:
>
> Has anyone encountered this before?
>
> I am running version 2.16.13 of jtalert, 2.2.2 of WSJT-x and 15.8.0 of Dxkeeper.
>
> W9PDS
>
>
> Has this always been a problem with your Flex and multi-instance JTAlert or is it a recent manifestation? If it is recent, what was the last version of JTAlert that didn't exhibit this behavior?
>
> de Laurie VK3AMA
>
>
>



--
Bill Barrett
352-437-4758






TomK
 

From the FWIW (For What It's Worth) depart.
Sorry, if you know this, it have help others understand CPU loading issues.

WSJT and JTA present X-Large and Large processor task loads.
For JTA alone, think about the high message rate to display, lookup, resolve, flag, audio alert ... whoa!

As processes consume PC cycles, other processes are obviously given less time.
Moreover, a keyboard is an interrupt-driven task.
When you click a key, interrupts are thrown, the BIOS passes it to the OS.
It then passes control to an interrupt handler which passes control to registered s/w handlers.
The registered interrupt handlers do their things then finish (consuming more CPU cycles).
Only at a logically high level, are interrupts "asynchronous".
The reality is that multiple interrupts may asynchronously trig interrupt handlers.
However, once those handlers start to execute, the same CPU's cycles are consumed for all tasks.
At each CPU's level, you have synchronous code streams.
Multicore CPUs can ameliorate that but their allocations are seldom what an app would like.
While all that is happening, the CPU is still being interrupted 6 ways from Sunday.
So, with all that intense backdrop, you still have intense CPU tasks to execute.

So, a keyboard should and will slow down when the CPU is already heavily occupied and can't process all the interrupts.
Add 2, 3, or more JTA's and/or WSJT and it all gets that much worse.

The fact that NOT running any JTA instances makes your k/b faster seems entirely logical and expected.

I only suspect empirically, without seeing the code, that JTA doesn't do a lot of k/b interrupts itself.

In short, it's bloody amazing that a PC can handle the intense FFT's in WSJT as well as all that JTA does.

Just my 13 cents
TomK KT1TK 73

-----Original Message-----
From: Support@HamApps.groups.io <Support@HamApps.groups.io> On Behalf Of Bill Barrett
Sent: Friday, September 25, 2020 7:42 AM
To: Support@hamapps.groups.io
Subject: Re: [HamApps] Keyboard latency with multiple JTAlert processes running.

Hello Laurie-

I have noticed this for a long time. Never knew what caused it. Always assumed it was due to running multiple instances of WSJT-X.
The reason Flex users notice it is because we have multiple slices and run multiple instances of WSJT-X.
The keyboard delay is definitely related to using JTAlert. Without JTAlert the keyboard response if fast.
In my case with the 6700 can run as many as 8 instances of "X" and JTAlert. Then the keyboard response seems stacked up.
When pressing down on the left or right arrow key may take a second before there is a response on the screen and then the cursor moves several positions.
Hard to tell where the interference is happening, however, seems like the keyboard interrupt handler is stalled.
Hope this helps.
Bill W2PKY


On Thu, Sep 24, 2020 at 10:44 PM HamApps Support (VK3AMA) <vk3ama.ham.apps@...> wrote:

On 23/09/2020 6:54 am, Patrick Skerrett wrote:

Has anyone encountered this before?

I am running version 2.16.13 of jtalert, 2.2.2 of WSJT-x and 15.8.0 of Dxkeeper.

W9PDS


Has this always been a problem with your Flex and multi-instance JTAlert or is it a recent manifestation? If it is recent, what was the last version of JTAlert that didn't exhibit this behavior?

de Laurie VK3AMA




--
Bill Barrett
352-437-4758


m_scot_alex
 

This isn't a 'miracle of computing' issue; I can totally saturate the CPU, the GPU and the PCI bus(es) in all of my systems, including the system that I use WSJT-X/JT Alert most often, with 3D modelling or synthetic benchmarks and I don't see this behavior. This is very specific to JT Alert - I can shut down the WSJT-X instances and still have this problem as long as JT Alert is running. I suspect that this is ecosystem-specific, and not everyone experiences this issue, but it's real and it's not a CPU overload issue.

Laurie - I think this a keyboard event trapping race condition caused by 'hotkey' overlays in other programs. I'm travelling and this difficult to isolate remotely because of remote access latency, but I reduced the keyboard latency on my main JT-mode system by stopping an Intel video 'hotkey' utility. I'll provide more data next week and can send you a support request if you think it would help.

73,

Mike - N8MSA


Bill Barrett
 

Tom-
The PC CPU is only stressed during the decode period. Otherwise the
CPU is 93% idle.
WSJT-X and JTA take almost no CPU power during the sampling process.
The keyboard slow down is noticed anytime.
I'm sure Laurie will determine the cause and correct the problem now
that it's been uncovered.
Since most users do not run more than one instance of JTA and "X" they
do not notice the problem.
Bill W2PKY

On Sat, Sep 26, 2020 at 1:57 AM TomK <@KT1TK> wrote:

From the FWIW (For What It's Worth) depart.
Sorry, if you know this, it have help others understand CPU loading issues.

WSJT and JTA present X-Large and Large processor task loads.
For JTA alone, think about the high message rate to display, lookup, resolve, flag, audio alert ... whoa!

As processes consume PC cycles, other processes are obviously given less time.
Moreover, a keyboard is an interrupt-driven task.
When you click a key, interrupts are thrown, the BIOS passes it to the OS.
It then passes control to an interrupt handler which passes control to registered s/w handlers.
The registered interrupt handlers do their things then finish (consuming more CPU cycles).
Only at a logically high level, are interrupts "asynchronous".
The reality is that multiple interrupts may asynchronously trig interrupt handlers.
However, once those handlers start to execute, the same CPU's cycles are consumed for all tasks.
At each CPU's level, you have synchronous code streams.
Multicore CPUs can ameliorate that but their allocations are seldom what an app would like.
While all that is happening, the CPU is still being interrupted 6 ways from Sunday.
So, with all that intense backdrop, you still have intense CPU tasks to execute.

So, a keyboard should and will slow down when the CPU is already heavily occupied and can't process all the interrupts.
Add 2, 3, or more JTA's and/or WSJT and it all gets that much worse.

The fact that NOT running any JTA instances makes your k/b faster seems entirely logical and expected.

I only suspect empirically, without seeing the code, that JTA doesn't do a lot of k/b interrupts itself.

In short, it's bloody amazing that a PC can handle the intense FFT's in WSJT as well as all that JTA does.

Just my 13 cents
TomK KT1TK 73


-----Original Message-----
From: Support@HamApps.groups.io <Support@HamApps.groups.io> On Behalf Of Bill Barrett
Sent: Friday, September 25, 2020 7:42 AM
To: Support@hamapps.groups.io
Subject: Re: [HamApps] Keyboard latency with multiple JTAlert processes running.

Hello Laurie-

I have noticed this for a long time. Never knew what caused it. Always assumed it was due to running multiple instances of WSJT-X.
The reason Flex users notice it is because we have multiple slices and run multiple instances of WSJT-X.
The keyboard delay is definitely related to using JTAlert. Without JTAlert the keyboard response if fast.
In my case with the 6700 can run as many as 8 instances of "X" and JTAlert. Then the keyboard response seems stacked up.
When pressing down on the left or right arrow key may take a second before there is a response on the screen and then the cursor moves several positions.
Hard to tell where the interference is happening, however, seems like the keyboard interrupt handler is stalled.
Hope this helps.
Bill W2PKY


On Thu, Sep 24, 2020 at 10:44 PM HamApps Support (VK3AMA) <vk3ama.ham.apps@...> wrote:

On 23/09/2020 6:54 am, Patrick Skerrett wrote:

Has anyone encountered this before?

I am running version 2.16.13 of jtalert, 2.2.2 of WSJT-x and 15.8.0 of Dxkeeper.

W9PDS


Has this always been a problem with your Flex and multi-instance JTAlert or is it a recent manifestation? If it is recent, what was the last version of JTAlert that didn't exhibit this behavior?

de Laurie VK3AMA




--
Bill Barrett
352-437-4758











--
Bill Barrett
352-437-4758