You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-27 - 20:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Aug 27, 2026 12:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1917247991021,3cyclictest0-21swapper/010:25:020
191724799845,67cyclictest1947111-21modprobe10:00:000
1917251998221,28cyclictest150-21pr/ttyS210:26:241
1917251997958,12cyclictest1989661-21ssh11:30:011
191725199783,60cyclictest1925945-21ssh09:15:031
1917251997625,16cyclictest1967877-21sh10:43:571
1917251997424,18cyclictest1947208-21date10:00:021
1917251997224,20cyclictest2018293-21sh12:30:281
1917247997167,2cyclictest0-21swapper/011:45:020
1917255997011,17cyclictest1926563-21ssh09:16:212
1917251997025,13cyclictest2002064-21ssh11:56:071
1917251997024,15cyclictest2017614-21ssh12:29:211
1917258996829,35cyclictest1982468-21systemd-run11:15:013
1917258996829,35cyclictest1982468-21systemd-run11:15:003
1917251996825,15cyclictest1981425-21sh11:12:341
1917251996825,15cyclictest1981425-21sh11:12:341
191724799681,37cyclictest131rcu_preempt10:34:370
1917258996727,35cyclictest3467602-21systemd-journal08:45:013
1917258996727,35cyclictest3467602-21systemd-journal08:45:013
1917251996725,11cyclictest0-21swapper/111:16:001
1917251996723,14cyclictest2000267-21sh11:52:201
1917251996717,16cyclictest2006029-21ssh12:04:461
191725899663,11cyclictest1920795-21cron08:20:003
1917251996623,16cyclictest1976839-21ssh11:02:521
1917251996617,19cyclictest2007573-21sh12:07:561
1917251996617,19cyclictest2007573-21sh12:07:561
1917247996644,16cyclictest3467602-21systemd-journal12:40:010
191724799662,23cyclictest1917976-21munin-run07:25:000
191725899655,51cyclictest1923400-21modprobe09:09:593
1917258996526,35cyclictest2008100-21sh12:09:143
1917258996526,35cyclictest2008100-21sh12:09:143
1917258996525,34cyclictest1987451-21latency_hist11:25:033
191725899652,19cyclictest1942437-21cat09:50:013
1917251996520,13cyclictest1930984-21ssh09:25:361
1917247996512,11cyclictest0-21swapper/007:19:590
1917258996411,46cyclictest0-21swapper/310:30:013
1917251996420,13cyclictest1965101-21ssh10:37:541
1917258996323,36cyclictest37-21ksoftirqd/309:37:003
1917255996314,10cyclictest30-21ksoftirqd/210:47:272
1917255996314,10cyclictest30-21ksoftirqd/210:47:262
1917251996319,15cyclictest1935879-21rm09:35:521
1917247996317,42cyclictest12-21ksoftirqd/012:26:520
191724799625,9cyclictest1922867-21cron09:00:010
191724799625,9cyclictest1922867-21cron09:00:000
1917247996225,32cyclictest1925994-21sed09:15:030
191725899612,20cyclictest311-21rs:main3
191725899612,17cyclictest1947778-21ssh10:01:043
1917258996121,36cyclictest3467562-21sshd07:19:073
1917258996121,35cyclictest37-21ksoftirqd/312:25:213
191725899611,14cyclictest2020218-21unin-run12:35:013
1917255996113,10cyclictest30-21ksoftirqd/212:20:022
1917251996145,13cyclictest1968747-21ssh10:45:251
1917251996145,13cyclictest1968747-21ssh10:45:251
1917251996124,10cyclictest1941383-21ssh09:47:401
1917247996129,28cyclictest12-21ksoftirqd/011:48:140
191725899602,17cyclictest3467602-21systemd-journal12:20:013
1917258996020,36cyclictest3467562-21sshd11:02:083
1917251996046,11cyclictest2011609-21ssh12:16:191
1917247996044,10cyclictest0-21swapper/009:40:010
1917258995932,23cyclictest1999025-21munin-plugin-st11:50:003
1917258995927,27cyclictest3467602-21systemd-journal08:55:033
1917258995927,27cyclictest3467602-21systemd-journal08:55:023
1917258995925,29cyclictest1947149-21sh10:00:013
1917258995921,34cyclictest3467602-21systemd-journal12:24:313
1917258995921,33cyclictest37-21ksoftirqd/309:41:433
191725899591,15cyclictest48-21kcompactd011:55:273
191725899591,15cyclictest1956590-21systemctl10:20:003
1917255995914,10cyclictest30-21ksoftirqd/210:09:072
1917255995911,11cyclictest30-21ksoftirqd/211:23:402
1917251995946,10cyclictest2020689-21ssh12:35:351
191724799592,13cyclictest1-21systemd09:55:010
191725899583,15cyclictest1926584-21ssh09:16:253
1917258995823,30cyclictest1922403-21sed08:50:013
191725899582,15cyclictest1996891-21latency_hist11:45:033
191725899581,17cyclictest2006159-21rm12:05:003
1917255995814,9cyclictest30-21ksoftirqd/212:02:442
1917255995814,7cyclictest30-21ksoftirqd/211:57:042
1917251995847,9cyclictest2014408-21ssh12:22:211
1917251995845,10cyclictest1993418-21ssh11:37:581
1917251995845,10cyclictest1949972-21ssh10:05:401
1917251995843,11cyclictest1918467-21cron07:35:001
1917251995840,15cyclictest1938427-21ssh09:41:181
1917251995840,14cyclictest1962388-21ssh10:31:591
1917251995823,6cyclictest1948668-21sh10:03:031
1917247995829,13cyclictest2007801-21ssh12:08:250
1917247995829,13cyclictest2007801-21ssh12:08:250
1917247995821,33cyclictest0-21swapper/011:50:350
1917247995820,9cyclictest12-21ksoftirqd/009:15:200
1917258995726,27cyclictest3467602-21systemd-journal12:35:033
191725899572,16cyclictest1992725-21ssh11:36:133
191725899571,51cyclictest275-21dbus-daemon08:10:013
191725899571,15cyclictest1987828-21ssh11:25:523
191725599579,10cyclictest30-21ksoftirqd/211:38:492
1917255995716,7cyclictest30-21ksoftirqd/210:53:362
1917251995745,9cyclictest1978562-21ssh11:06:221
1917251995743,12cyclictest1953843-21ssh10:14:171
1917251995743,11cyclictest1973996-21ssh10:56:481
1917251995743,11cyclictest1958012-21ssh10:22:501
1917251995742,12cyclictest1985964-21ssh11:22:021
1917251995741,13cyclictest1955060-21ssh10:16:301
1917251995741,12cyclictest1989964-21ssh11:30:171
1917251995740,13cyclictest1995180-21ssh11:41:301
1917251995740,13cyclictest1920022-21sshd08:03:081
1917251995738,15cyclictest1971126-21ssh10:50:281
1917251995735,18cyclictest1922385-21munin-run08:50:001
191724799579,30cyclictest16950irq/224-800000011:15:030
191724799579,30cyclictest16950irq/224-800000011:15:030
191724799576,48cyclictest16950irq/224-800000011:20:020
1917258995624,28cyclictest1969634-21sh10:47:273
1917258995624,28cyclictest1969634-21sh10:47:273
1917258995619,32cyclictest1919123-21latency_hist07:45:023
1917258995618,5cyclictest1951138-21sh10:08:273
1917258995616,5cyclictest1923411-21diskmemload10:30:563
1917258995614,5cyclictest1931974-21rm09:27:533
191725899561,16cyclictest1919401-21grep07:50:023
1917251995642,11cyclictest1927638-21ssh09:18:461
1917251995640,13cyclictest2009462-21ssh12:11:451
1917251995638,14cyclictest1942758-21ssh09:50:241
1917251995637,16cyclictest1998317-21ssh11:48:181
1917251995636,16cyclictest1930019-21ssh09:23:521
1917247995652,2cyclictest0-21swapper/010:10:020
191724799562,39cyclictest1929779-21ssh09:23:150
1917247995623,3cyclictest131rcu_preempt11:09:030
191725899553,16cyclictest1923992-21ssh09:10:433
191725899552,15cyclictest1958974-21grep10:25:003
191725899552,14cyclictest1983349-21ssh11:16:353
1917258995521,29cyclictest1921810-21sshd08:35:353
1917258995519,5cyclictest1971835-21kworker/u8:310:53:523
1917258995516,8cyclictest37-21ksoftirqd/311:07:413
1917255995513,9cyclictest30-21ksoftirqd/211:27:242
1917251995539,13cyclictest1920260-21sshd08:05:071
1917251995539,12cyclictest1933207-21ssh09:30:161
191724799553,17cyclictest2003776-21unin-run12:00:000
1917247995528,12cyclictest2012952-21ssh12:19:310
1917258995420,30cyclictest1923411-21diskmemload09:53:023
1917258995414,4cyclictest1990438-21sh11:31:223
1917258995413,6cyclictest0-21swapper/311:51:063
191725899541,17cyclictest1953962-21ssh10:14:323
1917255995411,8cyclictest30-21ksoftirqd/211:44:162
1917251995441,10cyclictest1920773-21cron08:17:001
1917251995439,10cyclictest1919275-21munin-run07:50:011
1917247995431,16cyclictest3467602-21systemd-journal12:15:000
1917247995427,11cyclictest1993186-21ssh11:37:210
1917247995427,11cyclictest1968367-21ssh10:45:010
1917247995426,24cyclictest1977318-21ssh11:03:550
1917247995424,14cyclictest1934323-21ssh09:32:440
191725899532,15cyclictest1968372-21munin-run10:45:003
1917258995321,28cyclictest1922767-21latency_hist08:55:023
1917258995321,27cyclictest2008713-21sh12:10:033
1917258995315,9cyclictest37-21ksoftirqd/310:35:163
1917258995313,4cyclictest1923411-21diskmemload09:24:203
1917255995325,7cyclictest30-21ksoftirqd/210:27:062
1917255995313,8cyclictest1946641-21ssh09:59:012
1917255995312,9cyclictest30-21ksoftirqd/212:39:432
1917251995341,9cyclictest1921102-21munin-run08:25:001
1917251995340,10cyclictest1918239-21munin-run07:30:021
1917251995340,10cyclictest1918239-21munin-run07:30:021
1917247995342,7cyclictest16950irq/224-800000010:40:010
1917247995335,15cyclictest16850irq/134-800000011:30:020
1917247995326,12cyclictest1991131-21ssh11:33:010
1917258995222,27cyclictest37-21ksoftirqd/310:58:463
191725599529,6cyclictest291rcuc/210:15:532
1917255995244,5cyclictest2003791-21kworker/2:012:26:562
1917255995212,8cyclictest30-21ksoftirqd/209:29:312
1917251995238,10cyclictest1919514-21munin-run07:55:011
1917251995238,10cyclictest1918773-21munin-run07:40:011
1917251995237,11cyclictest1919012-21munin-run07:45:001
191725899512,18cyclictest1918493-21munin-run07:35:013
191725899512,12cyclictest1920528-21modprobe08:13:093
191725599519,9cyclictest1965450-21ssh10:38:492
1917255995132,14cyclictest275-21dbus-daemon08:20:002
1917255995112,6cyclictest30-21ksoftirqd/209:52:322
191725599510,13cyclictest2000522-21ssh11:53:012
1917251995138,10cyclictest1923133-21sshd09:02:081
1917251995138,10cyclictest1917416-21sshd07:13:091
1917251995126,20cyclictest1917888-21latency_hist07:20:021
1917247995133,12cyclictest150-21pr/ttyS208:30:030
1917247995128,16cyclictest275-21dbus-daemon07:30:010
1917247995128,16cyclictest275-21dbus-daemon07:30:010
191725899507,10cyclictest1919753-21modprobe08:00:013
1917255995015,7cyclictest30-21ksoftirqd/212:09:282
1917255995015,7cyclictest30-21ksoftirqd/212:09:272
1917255995010,8cyclictest2009007-21ssh12:10:472
1917251995038,9cyclictest364-21snmpd07:23:191
1917247995028,7cyclictest1971131-21ssh10:50:290
1917247995027,8cyclictest2018600-21ssh12:31:090
1917247995023,3cyclictest0-21swapper/010:08:220
1917247995023,12cyclictest2005055-21ssh12:02:320
1917247995016,6cyclictest12-21ksoftirqd/010:47:520
1917247995016,6cyclictest12-21ksoftirqd/010:47:510
1917247995015,31cyclictest1987409-21latency_hist11:25:010
1917255994941,5cyclictest1916617-21kworker/2:307:55:032
1917255994910,8cyclictest30-21ksoftirqd/210:41:402
1917251994936,9cyclictest48-21kcompactd007:55:411
1917251994934,12cyclictest364-21snmpd08:37:551
1917247994941,5cyclictest16950irq/224-800000007:39:590
191724799492,40cyclictest1921843-21sh08:40:000
191724799492,13cyclictest1956586-21munin-run10:20:000
191725899482,15cyclictest3467602-21systemd-journal07:52:463
1917258994810,11cyclictest37-21ksoftirqd/307:15:013
191725599483,15cyclictest1917522-21cat07:15:012
1917255994812,6cyclictest131rcu_preempt11:04:082
1917255994811,8cyclictest30-21ksoftirqd/212:21:042
1917251994839,5cyclictest1922626-21munin-run08:55:001
1917251994832,11cyclictest1922108-21cron08:45:001
1917251994832,11cyclictest1922108-21cron08:45:001
1917251994820,23cyclictest1923371-21modprobe09:05:021
191724799484,6cyclictest1922640-21unin-run08:55:010
1917247994841,4cyclictest16850irq/134-800000009:10:000
191724799483,37cyclictest1942364-21cron09:50:010
191724799482,10cyclictest1920803-21modprobe08:20:000
1917247994815,30cyclictest12-21ksoftirqd/010:01:350
191724799481,27cyclictest131rcu_preempt10:58:250
1917255994739,5cyclictest1921098-21kworker/2:311:19:112
191725599473,13cyclictest364-21snmpd11:08:272
1917255994730,13cyclictest275-21dbus-daemon09:50:022
1917251994733,10cyclictest364-21snmpd08:30:251
1917247994726,16cyclictest275-21dbus-daemon08:05:000
191724799472,11cyclictest1920561-21unin-run08:15:010
191725899462,9cyclictest1918726-21modprobe07:39:593
1917258994611,9cyclictest37-21ksoftirqd/309:05:003
191725599469,6cyclictest30-21ksoftirqd/209:13:002
1917255994634,9cyclictest1922885-21systemctl09:00:012
1917255994634,9cyclictest1922885-21systemctl09:00:012
1917255994629,12cyclictest275-21dbus-daemon08:30:012
1917255994613,7cyclictest30-21ksoftirqd/210:31:302
1917251994633,9cyclictest364-21snmpd08:11:251
1917251994626,16cyclictest1921319-21modprobe08:28:081
1917247994624,19cyclictest12-21ksoftirqd/012:25:020
191725899453,11cyclictest1918329-21latency_hist07:30:023
191725899453,11cyclictest1918329-21latency_hist07:30:013
1917258994524,18cyclictest37-21ksoftirqd/308:25:013
191725899452,11cyclictest1921603-21unin-run08:35:023
1917255994530,9cyclictest1920315-21latency_hist08:10:022
1917255994529,10cyclictest48-21kcompactd007:30:102
1917247994541,2cyclictest0-21swapper/008:35:010
1917247994527,15cyclictest1932690-21ssh09:29:300
1917247994525,16cyclictest3467602-21systemd-journal09:04:070
1917247994520,6cyclictest12-21ksoftirqd/010:20:470
1917255994432,9cyclictest1919762-21cron08:00:012
1917251994432,9cyclictest364-21snmpd08:58:011
1917251994432,9cyclictest364-21snmpd08:58:001
191724799441,38cyclictest1938183-21ssh09:40:460
191725899432,16cyclictest1-21systemd08:05:013
191725599434,7cyclictest30-21ksoftirqd/207:27:202
191725599434,7cyclictest30-21ksoftirqd/207:27:192
1917255994335,5cyclictest1968386-21kworker/2:211:10:162
1917255994335,5cyclictest1968386-21kworker/2:211:10:152
1917255994334,5cyclictest1922636-21kworker/2:209:43:012
1917255994327,13cyclictest1997433-21ssh11:46:142
1917255994327,13cyclictest1975340-21ssh10:59:532
1917255994326,14cyclictest1921806-21sshd08:35:092
1917255994326,13cyclictest1947648-21ssh10:00:492
1917255994312,16cyclictest291rcuc/210:21:312
1917255994310,6cyclictest0-21swapper/211:30:042
1917247994320,16cyclictest275-21dbus-daemon08:25:010
191725899422,14cyclictest1921321-21sshd08:29:083
1917255994227,12cyclictest1936569-21ssh09:37:382
1917255994227,11cyclictest1952583-21ssh10:11:172
1917255994226,13cyclictest2018191-21ssh12:30:152
191725599421,31cyclictest1918196-21kworker/u8:107:40:022
1917255994210,8cyclictest0-21swapper/208:48:052
1917258994130,8cyclictest1917942-21sshd07:20:233
191725599416,5cyclictest131rcu_preempt07:25:022
1917255994128,10cyclictest30-21ksoftirqd/209:33:232
1917255994125,13cyclictest1929034-21ssh09:21:302
1917255994025,11cyclictest1923171-21munin-run09:05:002
1917255994025,10cyclictest1923433-21systemctl09:10:012
1917255994024,13cyclictest1921554-21sshd08:30:082
191724799402,13cyclictest1-21systemd08:46:090
191725599393,9cyclictest30-21ksoftirqd/208:40:022
191725599393,9cyclictest30-21ksoftirqd/208:40:012
1917255993926,10cyclictest364-21snmpd08:52:592
1917247993926,9cyclictest1919232-21sshd07:46:090
191725599385,7cyclictest30-21ksoftirqd/208:00:082
191725599381,8cyclictest0-21swapper/207:20:002
1917247993822,12cyclictest12-21ksoftirqd/007:10:180
1917255993723,10cyclictest48-21kcompactd008:10:412
1917255993714,17cyclictest1919005-21modprobe07:45:012
191724799371,33cyclictest0-21swapper/008:06:480
1917255993624,9cyclictest1919328-21cat07:50:012
1917251993617,15cyclictest1917377-21sed07:10:011
191724799362,6cyclictest12-21ksoftirqd/008:00:010
1917247993623,9cyclictest1918648-21sed07:35:020
191724799362,10cyclictest3467562-21sshd07:50:080
1917247993620,12cyclictest1922100-21sshd08:43:080
1917247993620,12cyclictest1922100-21sshd08:43:070
191724799351,11cyclictest1918995-21modprobe07:44:080
1917255993412,17cyclictest1917310-21cat07:10:022
1917255993017,9cyclictest364-21snmpd08:22:522
1917247992915,9cyclictest0-21swapper/007:10:020
1917258992212,7cyclictest1917267-21sed07:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional