You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-15 - 04:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Mon Jun 15, 2026 00:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2278248997557,10cyclictest2278250-21cat19:10:010
2278262997334,11cyclictest1-21systemd22:00:023
227825399734,4cyclictest131rcu_preempt22:00:031
2278253997327,6cyclictest309-21in:imuxsock00:30:001
227825899714,35cyclictest2278682-21kworker/u8:319:30:022
2278248997017,32cyclictest12-21ksoftirqd/000:05:000
2278258996925,11cyclictest275-21dbus-daemon00:15:012
2278258996925,11cyclictest275-21dbus-daemon00:15:012
2278258996524,11cyclictest2284011-21sed21:00:022
2278262996427,10cyclictest3467602-21systemd-journal22:55:003
2278262996326,9cyclictest37-21ksoftirqd/321:20:193
2278262996326,9cyclictest37-21ksoftirqd/321:20:193
2278258996228,10cyclictest30-21ksoftirqd/222:12:262
2278248996126,17cyclictest2328707-21ssh22:50:010
2278248996118,38cyclictest2355197-21unin-run23:50:010
2278248996039,8cyclictest131rcu_preempt21:30:020
2278248996039,8cyclictest131rcu_preempt21:30:010
2278248996032,21cyclictest275-21dbus-daemon20:05:010
2278248996032,21cyclictest275-21dbus-daemon20:05:000
2278248996028,16cyclictest2371744-21ssh00:27:100
2278248996018,37cyclictest2364098-21wc00:10:010
2278248995940,14cyclictest1-21systemd00:00:020
227824899593,3cyclictest131rcu_preempt23:00:020
2278248995928,14cyclictest2323503-21ssh22:38:160
227824899592,38cyclictest2375059-21cron00:35:000
227824899592,38cyclictest2375059-21cron00:35:000
2278248995829,18cyclictest12-21ksoftirqd/021:17:120
2278248995826,21cyclictest12-21ksoftirqd/021:37:500
2278253995729,15cyclictest275-21dbus-daemon21:25:011
2278253995729,15cyclictest275-21dbus-daemon21:25:001
227824899574,3cyclictest131rcu_preempt21:04:320
227824899573,17cyclictest120-21kswapd022:20:010
2278248995729,13cyclictest2337402-21ssh23:09:410
2278253995525,10cyclictest23-21ksoftirqd/122:18:351
2278248995545,7cyclictest275-21dbus-daemon23:39:590
2278248995531,18cyclictest2377293-21lockfile-create00:39:590
227824899551,38cyclictest2369049-21ssh00:21:020
227824899551,38cyclictest2369049-21ssh00:21:020
227825399541,3cyclictest131rcu_preempt00:20:001
227824899544,11cyclictest2308810-21modprobe22:05:000
227824899543,45cyclictest2282827-21kworker/u8:021:29:330
227824899543,17cyclictest2366427-21latency_hist00:15:020
227824899543,17cyclictest2366427-21latency_hist00:15:020
227826299531,4cyclictest131rcu_preempt00:30:023
2278258995345,4cyclictest2279793-21kworker/2:221:39:432
2278258995334,13cyclictest275-21dbus-daemon00:00:002
2278248995346,4cyclictest16850irq/134-800000022:15:020
2278248995343,6cyclictest12-21ksoftirqd/019:20:010
2278248995343,6cyclictest12-21ksoftirqd/019:20:000
2278248995322,15cyclictest2330449-21ssh22:53:500
227826299524,4cyclictest131rcu_preempt23:25:023
227826299522,4cyclictest131rcu_preempt00:20:003
227824899526,39cyclictest2291282-21latency_hist21:25:030
227824899526,39cyclictest2291282-21latency_hist21:25:030
227824899525,4cyclictest131rcu_preempt23:15:030
227824899525,42cyclictest2342263-21latency_hist23:20:020
2278248995231,18cyclictest2282864-21run-parts20:40:000
2278248995231,14cyclictest275-21dbus-daemon22:45:010
2278248995231,14cyclictest275-21dbus-daemon22:45:000
2278248995228,19cyclictest275-21dbus-daemon22:29:590
2278248995216,6cyclictest12-21ksoftirqd/000:20:000
227824899521,35cyclictest2298978-21ssh21:42:400
2278262995136,11cyclictest1-21systemd23:00:013
2278258995143,5cyclictest2284200-21kworker/2:322:32:322
227825899513,14cyclictest2338532-21ssh23:11:532
2278258995131,15cyclictest1-21systemd22:30:002
2278248995137,7cyclictest309-21in:imuxsock20:50:010
227824899513,23cyclictest2348725-21sort23:35:020
2278248995130,5cyclictest16850irq/134-800000023:53:380
2278248995125,16cyclictest275-21dbus-daemon19:40:010
2278248995120,12cyclictest12-21ksoftirqd/022:10:020
227826299504,3cyclictest131rcu_preempt22:30:003
2278258995042,5cyclictest2279793-21kworker/2:221:13:072
2278258995042,5cyclictest2279793-21kworker/2:221:13:072
2278258995041,5cyclictest2279793-21kworker/2:221:27:342
2278258995032,14cyclictest2284412-21cron21:10:002
227825899502,6cyclictest131rcu_preempt23:05:022
227825399503,3cyclictest131rcu_preempt21:40:011
2278248995023,12cyclictest2320849-21ssh22:32:020
227826299499,6cyclictest131rcu_preempt23:36:363
227826299496,4cyclictest131rcu_preempt23:50:023
227826299492,4cyclictest131rcu_preempt23:33:053
2278258994942,4cyclictest2284200-21kworker/2:321:54:012
2278258994941,5cyclictest2353461-21kworker/2:100:30:552
2278258994941,5cyclictest2353461-21kworker/2:100:30:552
2278258994941,5cyclictest2320196-21kworker/2:023:49:332
2278258994941,5cyclictest2279793-21kworker/2:221:30:202
2278258994941,5cyclictest2279793-21kworker/2:221:30:192
2278258994940,6cyclictest2353461-21kworker/2:100:07:382
227825899493,4cyclictest131rcu_preempt00:23:482
227825899493,4cyclictest131rcu_preempt00:23:482
2278258994932,13cyclictest1-21systemd23:55:012
227825399493,3cyclictest131rcu_preempt22:50:011
227825399492,4cyclictest131rcu_preempt23:15:011
227824899495,9cyclictest2353254-21latency_hist23:45:030
227824899494,11cyclictest2283632-21modprobe20:55:000
2278248994938,7cyclictest16850irq/134-800000022:21:320
2278248994925,18cyclictest275-21dbus-daemon19:15:000
227826299485,4cyclictest131rcu_preempt21:50:003
227826299485,3cyclictest131rcu_preempt22:15:003
2278262994834,11cyclictest2284341-21sed21:05:023
2278262994833,10cyclictest3467602-21systemd-journal20:00:003
227826299482,4cyclictest131rcu_preempt23:40:143
227826299482,4cyclictest131rcu_preempt00:40:003
227826299482,4cyclictest131rcu_preempt00:33:433
227826299482,4cyclictest131rcu_preempt00:33:433
2278262994810,6cyclictest131rcu_preempt20:59:593
2278258994840,5cyclictest2284200-21kworker/2:321:21:512
2278258994840,5cyclictest2284200-21kworker/2:321:21:502
2278258994840,5cyclictest2279793-21kworker/2:221:19:412
2278258994839,5cyclictest2284200-21kworker/2:322:06:352
227825399486,6cyclictest131rcu_preempt22:40:001
227825399482,3cyclictest131rcu_preempt23:40:001
2278248994835,8cyclictest2280767-21cron20:00:000
2278248994831,14cyclictest2304949-21ssh21:55:550
2278248994830,14cyclictest2302135-21unin-run21:50:000
2278248994829,15cyclictest2345239-21ssh23:26:570
227824899481,4cyclictest131rcu_preempt23:00:030
2278248994810,28cyclictest2284421-21modprobe21:10:000
227826299477,3cyclictest131rcu_preempt21:40:133
227826299475,3cyclictest131rcu_preempt21:40:003
227826299474,4cyclictest131rcu_preempt23:11:083
227826299474,4cyclictest131rcu_preempt22:19:593
227826299473,9cyclictest37-21ksoftirqd/300:10:023
227826299472,4cyclictest131rcu_preempt20:24:593
227826299472,4cyclictest131rcu_preempt00:21:133
227826299472,4cyclictest131rcu_preempt00:21:133
227825899474,3cyclictest131rcu_preempt00:16:472
2278258994740,4cyclictest2284200-21kworker/2:323:36:122
2278258994739,5cyclictest2320196-21kworker/2:023:32:072
2278258994739,5cyclictest2320196-21kworker/2:022:55:032
2278258994739,5cyclictest2284200-21kworker/2:322:21:492
2278258994739,5cyclictest2284200-21kworker/2:321:49:182
2278258994733,10cyclictest3467602-21systemd-journal20:31:342
2278258994733,10cyclictest3467602-21systemd-journal20:31:342
2278258994711,6cyclictest131rcu_preempt20:48:332
227825399474,4cyclictest131rcu_preempt21:35:011
227825399474,4cyclictest131rcu_preempt21:35:001
2278253994731,12cyclictest275-21dbus-daemon23:20:011
227825399472,23cyclictest2278682-21kworker/u8:319:25:001
227825399471,5cyclictest131rcu_preempt21:15:321
2278253994715,6cyclictest3467718-21lldpd21:45:501
227825399471,16cyclictest2376083-21ssh00:37:021
2278253994711,6cyclictest131rcu_preempt00:10:001
2278248994729,14cyclictest2302857-21ssh21:51:120
227826299464,4cyclictest131rcu_preempt23:56:073
227826299464,3cyclictest131rcu_preempt19:40:003
2278262994630,12cyclictest2324217-21readlink22:40:003
2278262994626,16cyclictest2304476-21sed21:55:023
227826299461,4cyclictest131rcu_preempt22:10:023
227826299461,3cyclictest131rcu_preempt21:17:513
2278258994639,4cyclictest2284200-21kworker/2:321:56:352
2278258994638,5cyclictest2320196-21kworker/2:023:28:312
2278258994638,5cyclictest2284200-21kworker/2:323:41:302
2278258994638,5cyclictest2284200-21kworker/2:323:05:022
2278258994637,5cyclictest2284200-21kworker/2:322:00:092
2278258994633,9cyclictest2314387-21sshd22:17:302
2278258994611,6cyclictest131rcu_preempt22:48:432
2278253994634,9cyclictest2282365-21systemctl20:30:011
2278253994631,11cyclictest1-21systemd20:10:001
227825399462,4cyclictest131rcu_preempt19:30:001
227825399462,3cyclictest131rcu_preempt00:15:021
227825399462,3cyclictest131rcu_preempt00:15:021
227825399461,3cyclictest131rcu_preempt21:45:011
2278248994623,18cyclictest3467602-21systemd-journal20:35:010
2278248994623,18cyclictest3467602-21systemd-journal20:35:000
2278248994621,20cyclictest275-21dbus-daemon19:35:000
227824899461,40cyclictest2285145-21ssh21:11:200
227824899461,40cyclictest2285145-21ssh21:11:200
227826299454,4cyclictest131rcu_preempt22:00:203
227826299453,4cyclictest131rcu_preempt22:43:273
227826299453,4cyclictest131rcu_preempt22:43:273
227826299453,3cyclictest131rcu_preempt23:50:403
227826299453,3cyclictest131rcu_preempt23:02:153
227826299452,4cyclictest131rcu_preempt21:15:013
227826299452,4cyclictest131rcu_preempt21:15:003
227826299452,3cyclictest131rcu_preempt00:13:003
227826299452,3cyclictest131rcu_preempt00:12:593
227826299451,5cyclictest131rcu_preempt22:24:513
227826299451,4cyclictest131rcu_preempt22:33:443
227825899454,3cyclictest131rcu_preempt20:55:012
2278258994538,4cyclictest2284200-21kworker/2:322:42:502
2278258994538,4cyclictest2284200-21kworker/2:322:42:502
2278258994537,5cyclictest2284200-21kworker/2:321:40:052
2278258994537,5cyclictest2278016-21kworker/2:119:20:032
2278258994537,5cyclictest2278016-21kworker/2:119:20:022
227825399458,6cyclictest131rcu_preempt21:05:011
227825399455,3cyclictest131rcu_preempt22:05:021
227825399452,3cyclictest131rcu_preempt20:05:011
227825399452,3cyclictest131rcu_preempt20:05:001
2278248994530,11cyclictest12-21ksoftirqd/019:55:000
2278248994529,11cyclictest2281827-21cat20:20:010
2278248994529,11cyclictest2281827-21cat20:20:000
227826299441,3cyclictest131rcu_preempt20:35:013
227826299441,3cyclictest131rcu_preempt20:35:003
227825899444,3cyclictest131rcu_preempt23:20:482
2278258994435,6cyclictest2284200-21kworker/2:323:20:022
2278258994434,7cyclictest2324215-21sh22:39:592
2278258994410,5cyclictest131rcu_preempt00:25:232
227825399447,5cyclictest131rcu_preempt00:00:101
227825399443,4cyclictest131rcu_preempt23:45:161
227825399443,3cyclictest131rcu_preempt00:32:371
227825399443,3cyclictest131rcu_preempt00:32:371
2278253994430,10cyclictest1-21systemd20:15:001
227825399442,4cyclictest131rcu_preempt23:45:011
227825399442,3cyclictest131rcu_preempt22:27:241
227825399442,23cyclictest0-21swapper/121:27:041
227825399442,23cyclictest0-21swapper/119:45:011
2278253994422,16cyclictest1-21systemd00:00:001
227825399441,4cyclictest131rcu_preempt23:05:331
227825399441,4cyclictest131rcu_preempt22:55:021
227825399441,3cyclictest131rcu_preempt21:10:021
227824899449,32cyclictest12-21ksoftirqd/023:24:370
227824899446,27cyclictest2283890-21modprobe21:00:000
227826299433,3cyclictest131rcu_preempt19:35:013
227826299432,4cyclictest131rcu_preempt20:44:593
227826299432,4cyclictest131rcu_preempt19:25:003
227826299432,3cyclictest131rcu_preempt21:27:473
227826299432,3cyclictest131rcu_preempt00:00:093
227826299431,3cyclictest131rcu_preempt23:07:283
227826299431,3cyclictest131rcu_preempt22:47:053
227826299431,3cyclictest131rcu_preempt20:46:333
227826299431,3cyclictest131rcu_preempt19:54:323
2278258994335,5cyclictest2353461-21kworker/2:100:01:052
2278258994334,6cyclictest2271866-21kworker/2:019:30:022
227825399433,3cyclictest131rcu_preempt22:45:001
227825399433,3cyclictest131rcu_preempt22:44:591
2278253994330,10cyclictest3467602-21systemd-journal20:56:021
227825399432,5cyclictest131rcu_preempt22:25:021
227825399432,4cyclictest131rcu_preempt21:15:021
227825399432,4cyclictest131rcu_preempt21:15:011
227825399431,4cyclictest131rcu_preempt23:35:001
227825399431,3cyclictest131rcu_preempt21:54:331
227826299424,5cyclictest131rcu_preempt21:10:013
227826299422,3cyclictest131rcu_preempt23:16:193
227826299421,4cyclictest131rcu_preempt23:29:463
227826299421,4cyclictest131rcu_preempt20:55:033
227826299421,3cyclictest131rcu_preempt20:15:013
227825399424,4cyclictest131rcu_preempt23:20:381
2278253994227,12cyclictest2278430-21cron19:15:001
227825399421,4cyclictest131rcu_preempt22:08:501
227826299417,6cyclictest131rcu_preempt21:30:093
227826299417,6cyclictest131rcu_preempt21:30:093
227826299412,3cyclictest131rcu_preempt19:15:003
227826299411,3cyclictest131rcu_preempt20:10:013
227826299411,3cyclictest131rcu_preempt20:05:023
227826299411,3cyclictest131rcu_preempt20:05:023
227826299411,3cyclictest131rcu_preempt19:30:023
227825899414,3cyclictest131rcu_preempt20:05:012
227825899414,3cyclictest131rcu_preempt20:05:012
227825899412,6cyclictest131rcu_preempt00:40:002
227825399414,3cyclictest131rcu_preempt23:30:011
227825399412,4cyclictest131rcu_preempt19:55:001
227825399412,3cyclictest131rcu_preempt22:15:021
227825399411,4cyclictest131rcu_preempt22:55:581
227825399411,3cyclictest131rcu_preempt22:32:311
227825399411,3cyclictest131rcu_preempt19:50:021
227825399411,3cyclictest131rcu_preempt00:20:091
227825399411,3cyclictest131rcu_preempt00:20:091
2278248994136,3cyclictest16950irq/224-800000020:15:000
227826299403,4cyclictest1-21systemd19:40:333
227826299403,3cyclictest131rcu_preempt19:50:023
2278262994024,13cyclictest3467602-21systemd-journal20:15:023
2278262994024,13cyclictest3467602-21systemd-journal20:15:023
227826299401,3cyclictest131rcu_preempt19:10:023
2278258994032,5cyclictest2320196-21kworker/2:022:58:212
2278258994029,8cyclictest3467602-21systemd-journal19:11:342
2278258994018,8cyclictest2282821-21sshd20:35:322
2278253994028,8cyclictest3467602-21systemd-journal23:01:331
227825399402,3cyclictest131rcu_preempt19:20:021
227825399402,3cyclictest131rcu_preempt19:20:011
227825399401,4cyclictest131rcu_preempt20:40:001
227824899402,8cyclictest2281473-21date20:10:020
227824899402,35cyclictest1-21systemd19:28:330
2278248994012,24cyclictest0-21swapper/020:26:180
227825899394,6cyclictest131rcu_preempt19:40:002
227825899392,5cyclictest275-21dbus-daemon20:15:012
2278258993924,11cyclictest2280777-21cron20:00:002
227825899392,3cyclictest131rcu_preempt20:45:022
2278258993923,11cyclictest3467602-21systemd-journal20:09:592
2278258993921,13cyclictest3467602-21systemd-journal19:50:002
2278253993925,10cyclictest3467602-21systemd-journal20:00:001
227825399392,3cyclictest131rcu_preempt19:35:011
227825399391,3cyclictest131rcu_preempt23:50:571
227824899398,25cyclictest2283131-21modprobe20:45:000
227824899395,23cyclictest2282103-21modprobe20:25:000
227825899382,8cyclictest2279009-21ls19:25:012
2278258993823,11cyclictest1-21systemd19:55:002
2278258993810,5cyclictest131rcu_preempt20:15:012
2278258993810,5cyclictest131rcu_preempt20:15:012
227825399383,4cyclictest131rcu_preempt20:50:331
227824899382,31cyclictest0-21swapper/019:50:020
2278248993816,12cyclictest309-21in:imuxsock19:45:000
227824899381,31cyclictest151rcuc/019:25:020
227826299374,3cyclictest131rcu_preempt19:20:023
227826299374,3cyclictest131rcu_preempt19:20:023
2278258993729,5cyclictest2271866-21kworker/2:019:10:002
2278258993714,6cyclictest131rcu_preempt21:05:022
2278253993728,5cyclictest309-21in:imuxsock19:39:591
2278253993726,8cyclictest0-21swapper/119:10:011
2278253993725,9cyclictest2281797-21sshd20:17:331
2278253993725,9cyclictest2281797-21sshd20:17:331
227825399371,4cyclictest131rcu_preempt20:35:011
227825399371,4cyclictest131rcu_preempt20:35:001
2278262993625,8cyclictest2282866-21systemctl20:40:003
2278258993625,8cyclictest2282327-21sshd20:27:102
227826299351,4cyclictest131rcu_preempt20:30:023
227825899354,3cyclictest131rcu_preempt19:45:022
2278253993525,7cyclictest2283264-21latency_hist20:45:021
2278253993524,8cyclictest275-21dbus-daemon20:24:591
2278258993320,10cyclictest30-21ksoftirqd/220:20:162
227825399334,4cyclictest131rcu_preempt20:50:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional