You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-08 - 19:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Wed Apr 08, 2026 12:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
186128399107100,4cyclictest0-21swapper/007:25:010
1861283999727,15cyclictest1889876-21ssh10:01:270
186128399921,40cyclictest1897681-21ssh10:19:480
186128399921,40cyclictest1897681-21ssh10:19:480
1861283999131,26cyclictest1944207-21ssh12:07:100
1861283999029,15cyclictest1940005-21ssh11:57:330
1861283999029,15cyclictest1940005-21ssh11:57:330
1861283998764,20cyclictest1921549-21ssh11:15:010
1861283998427,12cyclictest1956308-21ssh12:35:050
186128399822,10cyclictest1880407-21munin-run09:40:000
1861283998017,53cyclictest16850irq/134-800000009:30:410
186128399791,6cyclictest1928040-21munin-run11:30:010
186128399781,43cyclictest1907556-21ssh10:42:270
1861283997724,14cyclictest1950824-21ssh12:22:340
1861283997627,10cyclictest1871673-21ssh09:19:420
1861283997627,10cyclictest1871673-21ssh09:19:420
1861283997525,13cyclictest1909398-21ssh10:46:430
1861283997520,11cyclictest1935572-21ssh11:47:100
1861283997426,16cyclictest1886194-21ssh09:53:060
1861283997416,17cyclictest16850irq/134-800000009:25:430
1861283997333,36cyclictest1919490-21latency_hist11:10:020
1861283997325,44cyclictest16950irq/224-800000008:05:010
1861283997318,50cyclictest16850irq/134-800000010:27:140
1861283997316,17cyclictest16850irq/134-800000010:05:470
1861283997227,14cyclictest1934201-21ssh11:44:160
1861283997216,17cyclictest16850irq/134-800000011:24:570
1861283997211,5cyclictest16850irq/134-800000012:04:110
1861283997117,49cyclictest16850irq/134-800000011:33:200
1861283997117,48cyclictest0-21swapper/008:55:000
1861283997111,18cyclictest16850irq/134-800000012:29:490
1861283997024,12cyclictest1948624-21ssh12:17:360
186128399702,40cyclictest1894279-21ssh10:11:370
1861283997020,13cyclictest1913681-21ssh10:56:380
1861283997020,13cyclictest1913681-21ssh10:56:380
1861283997014,14cyclictest1873767-21ssh09:24:290
1861283996927,13cyclictest1911401-21ssh10:51:190
1861283996921,11cyclictest1898156-21ssh10:20:290
186128399691,9cyclictest1863553-21sshd07:51:270
186128599672,48cyclictest1878377-21ssh09:35:042
186128599671,14cyclictest1917091-21ssh11:04:462
186128399671,19cyclictest0-21swapper/009:58:350
1861283996620,10cyclictest1922324-21ssh11:16:290
1861283996616,45cyclictest16850irq/134-800000009:47:570
1861285996521,40cyclictest30-21ksoftirqd/209:55:042
1861285996521,40cyclictest1889789-21sh10:01:082
1861283996512,17cyclictest16850irq/134-800000009:40:270
186128599641,13cyclictest1879936-21ssh09:38:472
1861285996327,31cyclictest1874069-21latency_hist09:25:012
1861285996322,37cyclictest30-21ksoftirqd/210:57:142
1861285996322,37cyclictest30-21ksoftirqd/210:57:142
1861285996322,36cyclictest1924946-1kworker/u9:112:10:002
186128599632,11cyclictest1941991-21ssh12:01:582
186128599631,14cyclictest48-21kcompactd008:40:032
186128599631,14cyclictest48-21kcompactd008:40:032
1861283996324,11cyclictest1954667-21rm12:31:230
186128399631,8cyclictest1917216-21/usr/sbin/munin11:05:000
1861283996314,6cyclictest16850irq/134-800000011:35:070
186128399631,32cyclictest1938824-21sed11:55:010
186128399631,28cyclictest0-21swapper/010:38:580
1861289996210,21cyclictest131rcu_preempt08:05:013
186128599622,12cyclictest364-21snmpd10:42:002
1861285996214,44cyclictest1865679-21sh08:35:002
186128599621,15cyclictest48-21kcompactd012:15:302
186128599621,12cyclictest1894064-21ssh10:11:072
1861285996129,25cyclictest309-21in:imuxsock11:40:012
186128599612,17cyclictest1939930-21ssh11:57:232
186128599612,17cyclictest1939930-21ssh11:57:222
186128599612,16cyclictest1927038-21ssh11:27:302
1861285996114,43cyclictest3467602-21systemd-journal10:08:332
186128599611,14cyclictest1885388-21ssh09:51:042
186128599611,12cyclictest1900204-21ssh10:25:102
186128599611,11cyclictest1883005-21ssh09:45:342
186128399611,11cyclictest1869395-21modprobe09:14:270
1861289996038,17cyclictest3467602-21systemd-journal12:25:003
1861285996013,40cyclictest1866694-21cron08:55:002
186128599601,14cyclictest1865394-21sshd08:28:262
186128599601,14cyclictest1865394-21sshd08:28:262
186128599601,11cyclictest1909562-21ssh10:47:072
1861283996027,10cyclictest12-21ksoftirqd/012:10:050
186128599593,14cyclictest1912871-21munin-run10:55:022
1861285995926,28cyclictest1928423-21sh11:30:372
186128599592,16cyclictest1958275-21cat12:40:012
186128599592,13cyclictest364-21snmpd07:15:182
186128599592,11cyclictest364-21snmpd11:10:372
1861285995917,37cyclictest1902641-21rm10:30:542
186128599591,17cyclictest1955812-21sshd12:34:272
186128599591,16cyclictest1869153-21ssh09:13:522
186128599591,12cyclictest364-21snmpd11:48:392
1861289995834,19cyclictest275-21dbus-daemon09:30:013
186128599582,14cyclictest1937567-21ssh11:51:552
186128599582,12cyclictest364-21snmpd09:19:072
186128599582,12cyclictest364-21snmpd09:19:072
1861285995818,36cyclictest1951710-21sh12:24:562
186128599581,17cyclictest1922558-21ssh11:17:092
1861285995810,4cyclictest0-21swapper/211:41:352
186128599579,4cyclictest1896220-21ssh10:16:032
186128599579,4cyclictest1896220-21ssh10:16:022
186128599572,9cyclictest1918745-21modprobe11:08:272
186128599572,8cyclictest1861565-21grep07:15:022
186128599572,14cyclictest1862299-21munin-run07:30:012
186128599572,13cyclictest1875634-21ssh09:28:482
1861285995719,35cyclictest30-21ksoftirqd/207:25:002
1861285995718,34cyclictest1862538-21cron07:35:012
186128599571,12cyclictest364-21snmpd10:22:062
186128599571,10cyclictest1865958-21true08:40:022
1861283995713,38cyclictest0-21swapper/010:35:010
186128599562,8cyclictest1862789-21cron07:40:002
186128599562,17cyclictest1882559-21munin-run09:45:002
186128599562,13cyclictest1946949-21ssh12:13:522
186128599562,11cyclictest364-21snmpd08:47:242
186128599561,11cyclictest48-21kcompactd010:38:312
1861289995535,11cyclictest1864873-21cron08:20:003
1861289995530,18cyclictest275-21dbus-daemon10:45:013
186128599553,12cyclictest1867505-21ssh09:10:002
186128599552,13cyclictest1952312-21ssh12:25:522
186128599551,13cyclictest1863882-21munin-run08:00:012
186128599551,13cyclictest1863882-21munin-run08:00:012
186128599551,11cyclictest364-21snmpd08:06:222
1861284995539,12cyclictest275-21dbus-daemon09:10:011
1861289995443,9cyclictest37-21ksoftirqd/309:20:013
1861289995443,9cyclictest37-21ksoftirqd/309:20:013
1861289995433,15cyclictest1-21systemd10:55:013
186128599542,14cyclictest364-21snmpd08:57:542
186128599542,12cyclictest1865178-21unin-run08:25:012
186128599542,11cyclictest1925061-21ssh11:23:012
1861284995436,13cyclictest275-21dbus-daemon12:23:271
1861289995332,17cyclictest1-21systemd12:00:013
1861289995332,17cyclictest1-21systemd12:00:013
186128599532,11cyclictest1867256-21latency_hist09:05:022
186128599521,12cyclictest3467562-21sshd08:03:272
186128599521,12cyclictest1863291-21latency_hist07:45:032
1861283995234,13cyclictest1-21systemd08:40:000
1861283995222,13cyclictest1863308-21sshd07:48:230
1861289995134,12cyclictest275-21dbus-daemon09:45:003
1861285995110,4cyclictest1863270-21latency_hist07:45:022
1861289995046,2cyclictest0-21swapper/309:40:023
1861289995033,9cyclictest309-21in:imuxsock11:40:013
186128599502,13cyclictest1863349-21munin-run07:50:002
186128599501,13cyclictest1863569-21modprobe07:54:272
1861289994939,6cyclictest1866476-21latency_hist08:50:013
1861289994926,18cyclictest275-21dbus-daemon09:35:023
186128599492,13cyclictest1864929-21munin-run08:20:012
1861284994933,12cyclictest3467602-21systemd-journal09:05:011
1861289994835,9cyclictest1953941-21munin-plugin-st12:30:003
1861289994831,11cyclictest1867498-21munin-plugin-st09:10:003
1861284994829,13cyclictest275-21dbus-daemon11:40:011
1861283994831,11cyclictest275-21dbus-daemon07:39:590
186128399481,28cyclictest0-21swapper/008:32:430
186128999476,26cyclictest1897765-21modprobe10:20:013
186128999476,26cyclictest1897765-21modprobe10:20:003
1861284994734,9cyclictest309-21in:imuxsock08:20:001
1861284994711,5cyclictest131rcu_preempt10:15:561
1861284994711,5cyclictest131rcu_preempt10:15:561
1861289994626,14cyclictest309-21in:imuxsock11:15:003
1861289994623,18cyclictest275-21dbus-daemon09:00:013
1861289994622,5cyclictest37-21ksoftirqd/310:35:343
1861284994634,8cyclictest1923681-21cron11:20:001
1861284994612,6cyclictest131rcu_preempt10:53:401
1861289994526,12cyclictest309-21in:imuxsock11:30:013
1861285994535,5cyclictest1864623-21munin-plugin-st08:15:002
1861285994535,5cyclictest1864623-21munin-plugin-st08:15:002
1861284994533,9cyclictest23-21ksoftirqd/112:35:021
186128499452,13cyclictest1889614-21sh10:00:431
1861284994511,6cyclictest131rcu_preempt09:58:231
186128499451,13cyclictest1940728-21ssh11:59:281
186128499451,13cyclictest1940728-21ssh11:59:271
1861283994527,14cyclictest1864861-21sshd08:18:250
1861289994433,7cyclictest37-21ksoftirqd/309:15:033
1861289994428,11cyclictest309-21in:imuxsock11:40:033
186128999441,3cyclictest131rcu_preempt12:38:423
1861284994435,5cyclictest1871814-21munin-plugin-st09:20:001
1861284994435,5cyclictest1871814-21munin-plugin-st09:20:001
1861284994432,9cyclictest1910459-21sshd10:49:281
1861284994411,6cyclictest131rcu_preempt12:36:261
1861284994410,6cyclictest131rcu_preempt09:24:041
1861284994410,5cyclictest131rcu_preempt11:52:571
186128999434,25cyclictest1866205-21grep08:45:003
186128999434,25cyclictest1866205-21grep08:45:003
1861289994327,11cyclictest1-21systemd07:25:003
1861289994318,6cyclictest37-21ksoftirqd/309:55:353
186128499438,5cyclictest131rcu_preempt11:08:461
186128499433,3cyclictest131rcu_preempt11:28:001
1861284994324,15cyclictest275-21dbus-daemon11:05:001
1861284994311,5cyclictest131rcu_preempt10:41:101
1861284994310,5cyclictest131rcu_preempt11:13:321
1861283994329,9cyclictest309-21in:imuxsock07:30:010
186128399431,26cyclictest0-21swapper/007:15:300
1861289994229,8cyclictest1867495-21diskmemload12:15:023
1861289994218,5cyclictest37-21ksoftirqd/310:01:133
1861289994212,6cyclictest131rcu_preempt11:25:013
186128599422,11cyclictest1861417-21cat07:10:022
1861284994233,6cyclictest23-21ksoftirqd/109:45:021
1861284994230,7cyclictest310-21in:imklog09:45:381
1861284994225,12cyclictest3467602-21systemd-journal12:14:591
186128499422,3cyclictest131rcu_preempt11:40:301
186128499421,3cyclictest131rcu_preempt12:19:341
1861283994227,11cyclictest275-21dbus-daemon08:55:480
1861283994227,10cyclictest275-21dbus-daemon08:15:010
1861283994227,10cyclictest275-21dbus-daemon08:15:010
1861283994225,13cyclictest1865149-21sshd08:24:270
1861283994216,5cyclictest1867543-21latency_hist09:10:010
1861289994127,9cyclictest1862811-21systemctl07:40:003
1861289994124,13cyclictest1943511-21ssh12:05:233
1861289994120,14cyclictest275-21dbus-daemon08:00:013
1861289994120,14cyclictest275-21dbus-daemon08:00:013
1861289994111,6cyclictest131rcu_preempt09:55:013
186128499414,3cyclictest131rcu_preempt11:21:331
1861284994128,10cyclictest1866010-21wc08:40:011
186128499412,3cyclictest131rcu_preempt11:35:011
186128499412,3cyclictest131rcu_preempt09:15:021
186128399411,3cyclictest131rcu_preempt08:00:020
186128399411,3cyclictest131rcu_preempt08:00:020
186128399411,12cyclictest1861469-21sshd07:10:270
1861289994028,9cyclictest37-21ksoftirqd/310:34:053
1861289994028,6cyclictest37-21ksoftirqd/311:05:003
1861289994025,11cyclictest37-21ksoftirqd/309:21:553
1861289994021,15cyclictest1-21systemd11:00:013
1861289994021,15cyclictest1-21systemd11:00:013
1861289994019,16cyclictest1-21systemd07:20:013
186128499407,3cyclictest131rcu_preempt10:24:351
1861284994026,10cyclictest3467602-21systemd-journal08:25:001
1861284994025,12cyclictest23-21ksoftirqd/110:34:501
186128499402,4cyclictest131rcu_preempt10:12:561
1861284994024,12cyclictest275-21dbus-daemon07:45:011
1861284994024,12cyclictest275-21dbus-daemon07:45:011
1861283994022,12cyclictest275-21dbus-daemon08:45:010
1861283994022,12cyclictest275-21dbus-daemon08:45:010
186128399401,3cyclictest0-21swapper/007:30:060
186128399401,34cyclictest1863087-21true07:45:010
186128399401,34cyclictest1863087-21true07:45:010
186128999394,26cyclictest1865973-21grep08:40:013
1861289993924,8cyclictest0-21swapper/307:50:013
1861289993923,13cyclictest37-21ksoftirqd/312:15:233
1861289993913,19cyclictest131rcu_preempt10:25:013
1861289993910,4cyclictest131rcu_preempt11:45:083
1861284993931,4cyclictest1866705-21munin-plugin-st08:55:001
1861284993930,6cyclictest1906376-21latency_hist10:40:011
1861284993930,5cyclictest1952561-21sh12:26:281
1861284993925,11cyclictest23-21ksoftirqd/109:36:011
1861284993924,12cyclictest1866452-21latency_hist08:50:011
1861284993924,11cyclictest23-21ksoftirqd/110:06:221
1861284993923,11cyclictest275-21dbus-daemon09:00:011
1861283993925,11cyclictest3467602-21systemd-journal09:03:320
186128999387,22cyclictest1943128-21modprobe12:05:013
1861289993828,7cyclictest37-21ksoftirqd/310:29:113
1861289993826,10cyclictest37-21ksoftirqd/310:14:083
1861289993825,10cyclictest37-21ksoftirqd/307:15:023
1861289993824,11cyclictest37-21ksoftirqd/307:43:233
1861289993824,11cyclictest37-21ksoftirqd/307:43:233
1861289993822,11cyclictest37-21ksoftirqd/311:30:443
1861289993822,11cyclictest37-21ksoftirqd/310:45:143
1861289993820,12cyclictest275-21dbus-daemon11:55:003
1861289993816,15cyclictest275-21dbus-daemon09:05:013
186128499387,8cyclictest23-21ksoftirqd/109:25:171
186128499385,3cyclictest131rcu_preempt09:51:591
1861284993830,6cyclictest1943135-21munin-plugin-st12:05:001
1861284993828,7cyclictest1936472-21sshd11:49:261
1861284993826,9cyclictest1862467-21sed07:30:021
1861284993822,12cyclictest275-21dbus-daemon11:00:011
1861284993822,12cyclictest275-21dbus-daemon11:00:011
1861284993821,13cyclictest275-21dbus-daemon10:30:011
1861283993821,12cyclictest275-21dbus-daemon08:50:000
186128999378,5cyclictest131rcu_preempt11:18:463
186128999375,22cyclictest1919463-21ls11:10:023
1861289993725,7cyclictest1956041-21cron12:35:013
1861289993724,11cyclictest37-21ksoftirqd/309:50:013
186128999371,17cyclictest1855114-21kworker/u8:107:50:023
1861284993727,8cyclictest23-21ksoftirqd/109:31:501
1861284993726,8cyclictest23-21ksoftirqd/107:34:391
1861284993720,12cyclictest275-21dbus-daemon08:35:001
1861283993730,5cyclictest19350irq/34-mmc108:08:340
1861289993624,6cyclictest37-21ksoftirqd/308:30:003
1861289993624,6cyclictest37-21ksoftirqd/308:30:003
1861289993623,10cyclictest37-21ksoftirqd/310:05:273
1861289993621,12cyclictest37-21ksoftirqd/308:09:273
186128499367,9cyclictest0-21swapper/112:05:271
186128499364,9cyclictest23-21ksoftirqd/108:10:021
1861284993627,5cyclictest309-21in:imuxsock08:15:001
1861284993627,5cyclictest309-21in:imuxsock08:15:001
1861284993621,11cyclictest309-21in:imuxsock07:20:271
1861289993524,8cyclictest37-21ksoftirqd/307:35:033
1861289993522,9cyclictest3467602-21systemd-journal08:20:023
186128499355,3cyclictest131rcu_preempt07:20:031
1861284993523,8cyclictest23-21ksoftirqd/107:10:021
186128999345,21cyclictest1866700-21modprobe08:55:003
1861284993422,8cyclictest23-21ksoftirqd/107:50:001
1861284993421,9cyclictest23-21ksoftirqd/107:53:231
1861284993420,10cyclictest3467602-21systemd-journal08:00:001
1861284993420,10cyclictest3467602-21systemd-journal08:00:001
1861284993420,10cyclictest23-21ksoftirqd/107:40:011
1861284993417,13cyclictest23-21ksoftirqd/108:25:021
1861284993417,13cyclictest23-21ksoftirqd/108:25:021
1861284993410,6cyclictest131rcu_preempt08:05:001
1861289993323,5cyclictest37-21ksoftirqd/308:15:013
1861289993323,5cyclictest37-21ksoftirqd/308:15:013
1861283993321,9cyclictest275-21dbus-daemon08:30:020
1861283993321,9cyclictest275-21dbus-daemon08:30:020
186128399331,5cyclictest1861328-21sed07:10:010
1861289993221,6cyclictest37-21ksoftirqd/308:35:003
186128499322,3cyclictest131rcu_preempt07:10:011
186128999313,22cyclictest1-21systemd07:30:013
1861289993116,13cyclictest37-21ksoftirqd/307:10:013
1861284993121,7cyclictest1866356-21sed08:45:021
1861284993121,7cyclictest1866356-21sed08:45:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional