You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-27 - 08:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Sat Jun 27, 2026 00:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1918745999026,59cyclictest2006090-21ssh00:00:340
1918745998725,58cyclictest1970665-21ssh22:46:050
191874599861,81cyclictest1977384-21ssh23:00:040
191874599851,67cyclictest2002357-21ssh23:52:530
1918745998434,32cyclictest1933110-21ssh21:27:020
1918745998430,50cyclictest1992808-21ssh23:32:480
1918745998236,42cyclictest16950irq/224-800000022:30:180
1918745998025,51cyclictest2010689-21ssh00:10:120
1918745997927,35cyclictest2014114-21ssh00:17:380
1918745997826,35cyclictest0-21swapper/022:28:220
1918745997724,36cyclictest2009408-21ssh00:07:520
1918745997627,31cyclictest1950435-21ssh22:03:310
1918745997618,13cyclictest2023182-21ssh00:36:390
1918745997617,55cyclictest2020889-21ssh00:31:500
191874599761,57cyclictest0-21swapper/023:45:400
191874599761,57cyclictest0-21swapper/023:45:390
191874599755,46cyclictest16950irq/224-800000021:10:020
1918745997531,40cyclictest16950irq/224-800000022:20:000
1918745997528,29cyclictest1969729-21ssh22:44:250
1918745997428,39cyclictest1-21systemd22:40:000
1918745997424,45cyclictest1946807-21ssh21:55:390
1918745997424,33cyclictest1943709-21ssh21:49:280
1918745997418,12cyclictest1926885-21ssh21:14:140
191874599741,5cyclictest1924673-21munin-run21:05:010
1918745997327,29cyclictest1976899-21ssh22:59:250
1918745997326,30cyclictest1988704-21ssh23:24:120
1918745997325,44cyclictest2003805-21ssh23:55:440
1918745997325,43cyclictest1930038-21ssh21:20:260
1918745997321,46cyclictest1998527-21ssh23:44:590
1918745997319,49cyclictest16850irq/134-800000023:07:590
191874599731,68cyclictest0-21swapper/021:38:190
1918745997227,41cyclictest1945231-21ssh21:52:290
1918745997227,38cyclictest16950irq/224-800000019:40:010
1918745997223,2cyclictest0-21swapper/023:18:090
1918745997222,10cyclictest1935352-21ssh21:31:440
191874599701,64cyclictest1995981-21ssh23:39:390
191874599701,64cyclictest1995981-21ssh23:39:390
1918745997015,12cyclictest1972863-21ssh22:50:380
1918745996933,31cyclictest16850irq/134-800000000:30:020
1918760996817,11cyclictest37-21ksoftirqd/300:00:023
1918748996756,8cyclictest23-21ksoftirqd/100:00:031
1918745996719,41cyclictest0-21swapper/021:20:010
191874599663,17cyclictest1921264-21cron20:00:000
1918745996618,41cyclictest0-21swapper/020:55:010
1918745996618,41cyclictest0-21swapper/020:55:010
191874599661,51cyclictest0-21swapper/022:08:320
1918745996525,36cyclictest1990221-21sh23:27:140
1918745996523,11cyclictest2016394-21rm00:22:200
1918745996516,13cyclictest1955319-21ssh22:13:530
1918745996516,13cyclictest1955319-21ssh22:13:520
1918756996443,12cyclictest1943940-21kworker/2:223:29:502
1918745996425,35cyclictest1923080-21sshd20:32:310
1918745996425,35cyclictest1923080-21sshd20:32:300
1918745996425,34cyclictest0-21swapper/022:24:270
191876099628,10cyclictest37-21ksoftirqd/323:55:003
191875699624,10cyclictest30-21ksoftirqd/223:20:002
1918745996229,29cyclictest1982690-21ssh23:11:150
191876099614,9cyclictest37-21ksoftirqd/322:25:023
1918760996116,11cyclictest37-21ksoftirqd/323:20:023
191875699615,15cyclictest30-21ksoftirqd/222:15:022
191875699615,15cyclictest30-21ksoftirqd/222:15:012
1918756996113,5cyclictest131rcu_preempt23:00:012
191875699604,10cyclictest30-21ksoftirqd/221:25:002
1918756996020,11cyclictest30-21ksoftirqd/220:25:012
191874599602,18cyclictest1919212-21latency_hist19:19:590
191876099598,9cyclictest37-21ksoftirqd/321:25:023
191876099597,10cyclictest37-21ksoftirqd/321:50:023
191875699598,6cyclictest131rcu_preempt00:00:022
191875699592,9cyclictest30-21ksoftirqd/222:50:012
1918756995919,6cyclictest131rcu_preempt00:20:032
191876099584,10cyclictest37-21ksoftirqd/300:30:013
1918760995813,6cyclictest131rcu_preempt21:32:033
191875699589,6cyclictest131rcu_preempt22:55:012
191874599585,12cyclictest1922318-21modprobe20:20:000
191874599583,18cyclictest1923652-21unin-run20:45:010
191874599582,3cyclictest0-21swapper/021:43:420
1918745995822,4cyclictest1919988-21munin-plugin-st19:35:000
191876099577,9cyclictest37-21ksoftirqd/323:02:353
1918760995713,5cyclictest131rcu_preempt23:26:263
1918760995712,6cyclictest131rcu_preempt23:14:083
191875699576,12cyclictest30-21ksoftirqd/223:45:012
191875699575,10cyclictest30-21ksoftirqd/220:45:012
1918756995714,6cyclictest131rcu_preempt21:31:542
191874599573,11cyclictest1921809-21grep20:10:010
191874599573,11cyclictest1921809-21grep20:10:000
191876099567,9cyclictest37-21ksoftirqd/300:22:163
191876099566,8cyclictest37-21ksoftirqd/321:10:023
191876099566,11cyclictest37-21ksoftirqd/321:45:003
1918760995616,10cyclictest37-21ksoftirqd/322:17:313
1918760995613,6cyclictest131rcu_preempt22:41:333
1918760995612,6cyclictest131rcu_preempt23:45:113
1918760995612,6cyclictest131rcu_preempt23:45:103
1918760995611,6cyclictest131rcu_preempt22:52:023
191875699567,11cyclictest30-21ksoftirqd/220:15:012
191875699562,9cyclictest30-21ksoftirqd/200:10:022
1918756995614,5cyclictest131rcu_preempt21:47:392
1918756995611,6cyclictest131rcu_preempt21:40:002
191874599563,23cyclictest1921576-21cat20:05:010
191876099559,8cyclictest131rcu_preempt23:42:573
191876099559,5cyclictest131rcu_preempt21:15:063
191876099558,10cyclictest37-21ksoftirqd/300:15:343
191876099558,10cyclictest37-21ksoftirqd/300:07:593
1918760995517,9cyclictest37-21ksoftirqd/321:30:013
1918760995512,5cyclictest131rcu_preempt22:01:063
191875699556,9cyclictest30-21ksoftirqd/222:10:012
191875699556,6cyclictest131rcu_preempt00:04:192
1918756995513,6cyclictest131rcu_preempt21:43:452
1918756995512,6cyclictest131rcu_preempt21:29:352
191874899552,23cyclictest364-21snmpd22:40:031
1918745995512,8cyclictest0-21swapper/019:50:000
191876099547,9cyclictest37-21ksoftirqd/300:01:543
191876099547,8cyclictest37-21ksoftirqd/323:36:583
191876099547,8cyclictest37-21ksoftirqd/323:36:573
191876099547,11cyclictest37-21ksoftirqd/322:37:553
1918760995420,6cyclictest131rcu_preempt21:54:313
191876099541,9cyclictest1919487-21munin-run19:25:003
1918760995412,6cyclictest131rcu_preempt22:30:003
1918760995412,5cyclictest131rcu_preempt22:48:233
1918760995412,5cyclictest131rcu_preempt00:11:383
1918760995411,6cyclictest131rcu_preempt22:11:473
1918760995411,6cyclictest131rcu_preempt22:11:473
1918760995411,6cyclictest131rcu_preempt21:58:283
1918760995410,6cyclictest131rcu_preempt21:37:073
191875699549,5cyclictest131rcu_preempt00:13:542
1918756995414,6cyclictest131rcu_preempt21:50:382
1918756995411,6cyclictest131rcu_preempt22:35:022
1918745995425,25cyclictest1923358-21latency_hist20:40:010
1918745995418,4cyclictest2639831-1kworker/0:2H20:50:040
191876099533,10cyclictest37-21ksoftirqd/323:31:303
1918760995313,6cyclictest131rcu_preempt00:35:583
1918760995313,5cyclictest131rcu_preempt20:35:023
1918760995313,5cyclictest131rcu_preempt20:35:013
1918760995311,5cyclictest131rcu_preempt22:09:563
1918760995310,5cyclictest131rcu_preempt22:56:503
191875699536,7cyclictest30-21ksoftirqd/223:50:012
191875699536,7cyclictest30-21ksoftirqd/223:50:002
1918756995314,6cyclictest131rcu_preempt22:05:002
1918756995313,6cyclictest131rcu_preempt23:55:002
1918756995312,7cyclictest131rcu_preempt00:30:022
191876099524,16cyclictest37-21ksoftirqd/322:32:453
191875699529,6cyclictest131rcu_preempt00:31:222
191875699528,8cyclictest30-21ksoftirqd/200:20:332
191875699528,11cyclictest30-21ksoftirqd/221:20:012
191875699527,11cyclictest30-21ksoftirqd/220:59:592
191875699525,9cyclictest30-21ksoftirqd/223:34:282
1918756995244,5cyclictest1948732-21kworker/2:123:00:292
1918756995244,5cyclictest1948732-21kworker/2:122:17:222
1918756995244,5cyclictest1943940-21kworker/2:223:13:552
191875699523,8cyclictest30-21ksoftirqd/222:21:192
1918756995212,6cyclictest131rcu_preempt23:08:042
1918756995212,5cyclictest131rcu_preempt22:29:012
1918756995211,5cyclictest131rcu_preempt23:23:482
1918748995236,11cyclictest1-21systemd00:40:021
1918745995217,6cyclictest309-21in:imuxsock19:10:310
191876099519,6cyclictest131rcu_preempt00:33:043
191876099517,9cyclictest37-21ksoftirqd/323:06:383
191876099517,11cyclictest37-21ksoftirqd/319:45:013
191876099516,8cyclictest37-21ksoftirqd/320:20:013
191876099516,7cyclictest37-21ksoftirqd/320:45:023
1918760995110,5cyclictest131rcu_preempt21:11:153
191875699519,5cyclictest131rcu_preempt22:42:462
191875699514,8cyclictest30-21ksoftirqd/221:59:242
1918756995144,4cyclictest1920203-21kworker/2:120:50:252
1918756995144,4cyclictest1920203-21kworker/2:120:50:252
1918756995143,5cyclictest1948732-21kworker/2:100:37:312
1918756995113,5cyclictest131rcu_preempt19:45:012
1918756995110,6cyclictest131rcu_preempt23:36:122
1918756995110,6cyclictest131rcu_preempt23:36:112
1918748995130,10cyclictest23-21ksoftirqd/122:00:001
191874599513,7cyclictest3467602-21systemd-journal19:45:000
191876099508,9cyclictest37-21ksoftirqd/320:59:593
1918760995012,5cyclictest131rcu_preempt23:20:473
191875699509,6cyclictest131rcu_preempt20:05:022
1918756995043,4cyclictest1918737-21kworker/2:019:54:322
1918756995043,4cyclictest1910066-21kworker/2:319:19:322
1918756995042,5cyclictest1948732-21kworker/2:122:36:212
1918756995042,5cyclictest1920203-21kworker/2:121:13:482
1918756995042,5cyclictest1918737-21kworker/2:019:35:022
1918756995042,5cyclictest1918338-21kworker/2:119:10:252
1918756995012,6cyclictest131rcu_preempt19:25:012
191874899503,6cyclictest131rcu_preempt21:30:011
1918748995031,15cyclictest3467602-21systemd-journal20:20:031
191874599503,15cyclictest1921125-21cat19:55:030
191876099496,7cyclictest37-21ksoftirqd/319:55:003
191876099493,13cyclictest37-21ksoftirqd/319:40:003
191876099491,7cyclictest1921806-21munin-plugin-st20:10:003
191876099491,7cyclictest1921806-21munin-plugin-st20:10:003
191876099491,10cyclictest37-21ksoftirqd/320:05:003
1918756994941,5cyclictest1920203-21kworker/2:119:59:292
1918748994932,12cyclictest1921283-21systemctl20:00:011
1918748994920,21cyclictest23-21ksoftirqd/121:41:191
191874599494,10cyclictest1924405-21grep20:59:590
1918745994914,10cyclictest12-21ksoftirqd/020:30:010
191876099485,9cyclictest37-21ksoftirqd/321:05:013
1918760994814,5cyclictest131rcu_preempt19:15:023
1918756994841,4cyclictest1920203-21kworker/2:121:00:552
1918756994840,5cyclictest1921330-21kworker/2:320:05:522
1918756994840,5cyclictest1921330-21kworker/2:320:05:522
1918756994840,5cyclictest1920203-21kworker/2:121:09:232
1918756994840,5cyclictest1920203-21kworker/2:120:32:502
1918756994840,5cyclictest1920203-21kworker/2:120:32:502
1918756994840,5cyclictest1920203-21kworker/2:119:45:582
1918756994839,5cyclictest1921330-21kworker/2:320:15:502
1918748994825,17cyclictest3467602-21systemd-journal21:25:011
1918745994819,5cyclictest1919642-21latency_hist19:25:030
191874599481,8cyclictest1922262-21cat20:15:020
191876099473,11cyclictest37-21ksoftirqd/320:00:023
191876099472,8cyclictest37-21ksoftirqd/320:15:013
191876099472,8cyclictest37-21ksoftirqd/319:35:013
1918756994740,4cyclictest1920203-21kworker/2:120:27:062
1918756994739,5cyclictest1920203-21kworker/2:120:49:432
1918756994739,5cyclictest1920203-21kworker/2:120:37:072
1918756994739,5cyclictest1918737-21kworker/2:019:27:442
1918748994720,18cyclictest23-21ksoftirqd/100:25:001
1918748994713,6cyclictest131rcu_preempt23:50:021
1918748994713,6cyclictest131rcu_preempt23:50:011
1918745994713,31cyclictest12-21ksoftirqd/020:24:310
191876099461,19cyclictest0-21swapper/319:26:353
1918756994639,5cyclictest1918737-21kworker/2:019:36:422
1918748994629,12cyclictest23-21ksoftirqd/121:50:021
191874899462,8cyclictest0-21swapper/123:55:571
1918748994624,10cyclictest23-21ksoftirqd/122:15:011
1918748994624,10cyclictest23-21ksoftirqd/122:15:001
1918748994621,21cyclictest23-21ksoftirqd/121:19:301
1918748994619,22cyclictest23-21ksoftirqd/122:08:021
191876099451,9cyclictest284-21systemd-logind19:20:003
1918748994529,11cyclictest1939213-21systemctl21:40:011
1918748994526,7cyclictest23-21ksoftirqd/100:15:001
191874899451,10cyclictest0-21swapper/123:41:021
191876099447,9cyclictest37-21ksoftirqd/320:55:023
191876099447,9cyclictest37-21ksoftirqd/320:55:023
191876099445,9cyclictest37-21ksoftirqd/320:49:313
191876099441,11cyclictest37-21ksoftirqd/320:25:003
1918748994429,11cyclictest3467602-21systemd-journal19:25:001
1918748994428,8cyclictest23-21ksoftirqd/120:35:011
1918748994428,8cyclictest23-21ksoftirqd/120:35:001
1918748994427,7cyclictest23-21ksoftirqd/119:35:011
1918748994427,14cyclictest275-21dbus-daemon19:30:021
1918748994427,12cyclictest275-21dbus-daemon22:30:011
1918748994425,14cyclictest275-21dbus-daemon22:35:011
1918748994423,11cyclictest23-21ksoftirqd/121:33:341
1918748994420,20cyclictest23-21ksoftirqd/100:28:571
191874899441,5cyclictest0-21swapper/120:03:311
191874899441,5cyclictest0-21swapper/100:05:191
1918748994413,18cyclictest23-21ksoftirqd/100:19:161
1918748994411,17cyclictest23-21ksoftirqd/123:14:261
191876099434,8cyclictest37-21ksoftirqd/320:25:533
1918760994324,14cyclictest1-21systemd20:40:003
1918748994318,21cyclictest23-21ksoftirqd/123:31:241
191874899431,7cyclictest0-21swapper/123:15:501
191874899431,5cyclictest131rcu_preempt23:04:391
191874899431,5cyclictest0-21swapper/123:50:381
1918748994315,19cyclictest23-21ksoftirqd/123:20:431
191876099427,6cyclictest131rcu_preempt19:50:023
191874899426,5cyclictest131rcu_preempt22:01:081
191874899424,6cyclictest131rcu_preempt21:15:011
1918748994231,7cyclictest1920501-21sshd19:43:321
191874899421,8cyclictest0-21swapper/122:18:091
191874899421,5cyclictest0-21swapper/122:54:381
191874899419,5cyclictest131rcu_preempt19:49:251
191874899411,8cyclictest0-21swapper/122:46:281
191874899411,8cyclictest0-21swapper/122:21:151
191874899411,7cyclictest1923866-21sshd20:47:321
191874899411,7cyclictest0-21swapper/122:56:221
1918748994117,15cyclictest23-21ksoftirqd/123:38:561
1918748994117,15cyclictest23-21ksoftirqd/123:38:551
1918748994114,18cyclictest0-21swapper/123:27:081
1918748994029,7cyclictest23-21ksoftirqd/121:10:011
1918748994023,12cyclictest1924379-21cron21:00:001
191874899401,9cyclictest0-21swapper/100:30:491
191874899401,4cyclictest131rcu_preempt22:41:081
191876099393,7cyclictest37-21ksoftirqd/319:10:013
191874899397,6cyclictest131rcu_preempt20:50:061
191874899397,6cyclictest131rcu_preempt20:50:061
1918748993919,15cyclictest3467602-21systemd-journal23:05:031
191874899391,4cyclictest0-21swapper/121:52:111
1918745993927,9cyclictest1-21systemd19:30:010
1918756993825,10cyclictest1918755-21wc19:10:022
191874899388,5cyclictest131rcu_preempt20:39:591
191874899386,10cyclictest23-21ksoftirqd/120:42:531
1918748993727,6cyclictest309-21in:imuxsock19:20:011
1918748993724,9cyclictest23-21ksoftirqd/120:15:001
1918748993724,8cyclictest1920999-21sshd19:52:311
1918748993620,12cyclictest275-21dbus-daemon21:05:001
191874899361,6cyclictest0-21swapper/119:38:311
191874599366,4cyclictest131rcu_preempt19:10:010
1918748993424,7cyclictest1918757-21latency_hist19:10:011
1918748993416,13cyclictest275-21dbus-daemon20:30:001
191874899341,5cyclictest131rcu_preempt20:15:311
191874899341,5cyclictest0-21swapper/119:11:321
191874899331,5cyclictest131rcu_preempt20:06:321
191874899331,5cyclictest131rcu_preempt20:06:311
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional