You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-28 - 20:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sun Jun 28, 2026 12:45:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24088639910094,3cyclictest0-21swapper/011:45:020
240886699909,68cyclictest2475875-21ssh11:20:023
240886699909,68cyclictest2475875-21ssh11:20:013
2408863998626,15cyclictest2474603-21ssh11:17:170
2408863998626,15cyclictest2474603-21ssh11:17:170
240886399852,16cyclictest2445344-21munin-run10:15:000
2408863998430,49cyclictest16850irq/134-800000011:25:000
2408863998427,14cyclictest2490401-21ssh11:50:500
240886399823,14cyclictest2452212-21ssh10:29:500
2408863998123,13cyclictest2449876-21ssh10:24:370
2408863997933,41cyclictest16850irq/134-800000009:50:000
2408866997829,11cyclictest131rcu_preempt10:00:063
2408866997829,11cyclictest131rcu_preempt10:00:053
240886399782,6cyclictest2415110-21grep09:10:020
2408863997728,45cyclictest16950irq/224-800000010:00:010
2408863997728,45cyclictest16950irq/224-800000010:00:010
2408863997721,13cyclictest2462339-21ssh10:50:580
2408863997721,13cyclictest2462339-21ssh10:50:580
2408863997672,2cyclictest0-21swapper/008:40:030
2408863997672,2cyclictest0-21swapper/008:40:020
240886399762,8cyclictest2409321-21munin-run07:20:010
240886399762,8cyclictest2409321-21munin-run07:20:000
2408863997627,15cyclictest2447678-21ssh10:20:000
2408863997625,4cyclictest16950irq/224-800000010:05:000
2408863997521,13cyclictest2457866-21ssh10:41:290
2408863997521,13cyclictest2457866-21ssh10:41:290
2408863997427,12cyclictest2504348-21ssh12:20:360
2408863997325,15cyclictest2502127-21ssh12:15:570
2408863997325,15cyclictest2502127-21ssh12:15:560
2408863997323,10cyclictest2454947-21ssh10:35:060
2408865997212,5cyclictest131rcu_preempt10:10:022
2408865997212,5cyclictest131rcu_preempt10:10:012
2408863997225,8cyclictest2506746-21ssh12:25:390
2408863997225,8cyclictest2506746-21ssh12:25:380
2408863997129,36cyclictest0-21swapper/012:40:000
2408863997128,16cyclictest2425347-21ssh09:31:430
2408863997126,13cyclictest2497360-21ssh12:05:410
2408863997126,12cyclictest2471494-21ssh11:10:190
2408863997121,11cyclictest2434323-21ssh09:50:520
2408863997121,11cyclictest2434323-21ssh09:50:510
2408863997022,10cyclictest2469269-21ssh11:05:300
240886399701,26cyclictest2453240-21ssh10:31:400
2408863996926,12cyclictest2483007-21ssh11:35:060
2408863996922,13cyclictest2427194-21ssh09:35:410
2408863996920,14cyclictest2423899-21ssh09:28:520
2408863996917,11cyclictest2481818-21ssh11:32:440
240886399683,22cyclictest3467602-21systemd-journal09:20:000
2408863996824,12cyclictest2468549-21ssh11:04:290
2408863996824,12cyclictest2468549-21ssh11:04:280
2408863996818,12cyclictest2421370-21ssh09:23:210
240886399681,2cyclictest0-21swapper/007:15:020
2408864996722,12cyclictest23-21ksoftirqd/109:10:001
2408863996729,33cyclictest2464308-21ssh10:55:070
2408863996718,44cyclictest16850irq/134-800000012:31:590
2408864996516,9cyclictest23-21ksoftirqd/109:05:011
2408863996536,25cyclictest16850irq/134-800000011:50:010
2408865996439,6cyclictest2400427-21kworker/2:212:09:152
240886499647,10cyclictest23-21ksoftirqd/107:30:011
2408863996428,32cyclictest2416365-21sh09:12:280
2408864996321,11cyclictest23-21ksoftirqd/111:50:001
2408864996318,9cyclictest23-21ksoftirqd/110:29:141
2408863996359,2cyclictest0-21swapper/011:30:030
2408863996334,24cyclictest16950irq/224-800000012:13:260
2408863996321,2cyclictest0-21swapper/010:49:000
2408863996316,13cyclictest3467602-21systemd-journal09:45:000
2408865996238,7cyclictest2429133-21kworker/2:010:35:182
2408864996213,11cyclictest23-21ksoftirqd/111:10:131
2408863996258,2cyclictest0-21swapper/008:15:020
240886399622,29cyclictest2494767-21latency_hist12:00:030
2408865996139,7cyclictest2429133-21kworker/2:012:16:162
2408865996139,7cyclictest2429133-21kworker/2:012:16:152
2408864996116,11cyclictest23-21ksoftirqd/110:55:021
2408864996116,11cyclictest23-21ksoftirqd/110:55:011
2408863996121,10cyclictest2495171-21ssh12:01:040
2408863996111,17cyclictest16850irq/134-800000010:07:540
2408863996111,17cyclictest16850irq/134-800000010:07:530
2408865996040,7cyclictest2400427-21kworker/2:212:38:112
2408865996024,25cyclictest30-21ksoftirqd/210:00:062
2408865996024,25cyclictest30-21ksoftirqd/210:00:052
2408864996019,11cyclictest23-21ksoftirqd/112:20:011
2408864996019,11cyclictest23-21ksoftirqd/112:20:011
2408864996016,13cyclictest23-21ksoftirqd/111:25:021
240886599592,9cyclictest30-21ksoftirqd/209:25:022
2408864995919,10cyclictest23-21ksoftirqd/112:30:031
2408864995919,10cyclictest23-21ksoftirqd/112:30:021
2408864995915,10cyclictest23-21ksoftirqd/110:56:551
240886599586,8cyclictest30-21ksoftirqd/210:55:022
240886599586,8cyclictest30-21ksoftirqd/210:55:012
2408865995840,5cyclictest2429133-21kworker/2:012:30:222
2408864995811,13cyclictest23-21ksoftirqd/111:16:461
2408864995811,13cyclictest23-21ksoftirqd/111:16:451
240886699575,23cyclictest37-21ksoftirqd/308:40:023
240886699575,23cyclictest37-21ksoftirqd/308:40:023
2408866995749,5cyclictest37-21ksoftirqd/311:55:313
240886599578,6cyclictest131rcu_preempt10:35:022
240886499576,11cyclictest23-21ksoftirqd/109:52:081
240886499576,11cyclictest23-21ksoftirqd/109:52:071
2408864995717,11cyclictest23-21ksoftirqd/112:10:011
2408864995715,10cyclictest23-21ksoftirqd/109:48:001
240886599563,11cyclictest30-21ksoftirqd/212:00:012
2408865995621,11cyclictest30-21ksoftirqd/211:35:002
240886499567,7cyclictest23-21ksoftirqd/109:26:461
240886499566,7cyclictest23-21ksoftirqd/109:43:241
240886499565,15cyclictest23-21ksoftirqd/109:14:361
240886499562,14cyclictest23-21ksoftirqd/110:15:011
2408864995619,10cyclictest23-21ksoftirqd/109:35:001
2408864995612,14cyclictest23-21ksoftirqd/111:31:071
2408865995546,5cyclictest291rcuc/211:24:592
2408865995534,15cyclictest1-21systemd09:20:002
2408865995513,6cyclictest131rcu_preempt10:10:112
2408865995512,14cyclictest30-21ksoftirqd/209:14:552
240886499559,8cyclictest23-21ksoftirqd/108:40:001
240886499559,8cyclictest23-21ksoftirqd/108:40:001
240886499557,8cyclictest23-21ksoftirqd/112:40:021
240886499556,9cyclictest23-21ksoftirqd/109:40:011
240886499556,7cyclictest23-21ksoftirqd/112:10:571
240886499552,13cyclictest23-21ksoftirqd/107:40:011
2408864995521,12cyclictest23-21ksoftirqd/110:50:021
2408864995515,10cyclictest23-21ksoftirqd/111:02:041
2408864995515,10cyclictest23-21ksoftirqd/111:02:031
2408865995445,6cyclictest2400427-21kworker/2:212:23:522
240886599542,6cyclictest131rcu_preempt11:00:012
2408865995424,7cyclictest30-21ksoftirqd/209:25:032
2408865995414,7cyclictest131rcu_preempt10:29:162
2408865995413,16cyclictest131rcu_preempt12:01:212
240886499548,10cyclictest23-21ksoftirqd/111:09:281
240886499546,9cyclictest23-21ksoftirqd/112:23:151
240886499546,10cyclictest23-21ksoftirqd/110:37:551
240886499543,17cyclictest23-21ksoftirqd/111:52:031
2408866995345,6cyclictest37-21ksoftirqd/308:05:013
240886599539,12cyclictest30-21ksoftirqd/210:20:002
240886599538,10cyclictest30-21ksoftirqd/211:10:012
240886599536,6cyclictest131rcu_preempt11:55:012
2408865995327,8cyclictest30-21ksoftirqd/212:15:032
2408865995323,7cyclictest30-21ksoftirqd/207:25:012
2408865995312,5cyclictest131rcu_preempt09:35:022
2408865995311,6cyclictest131rcu_preempt12:28:242
2408865995311,6cyclictest131rcu_preempt12:28:232
2408865995311,5cyclictest131rcu_preempt11:18:522
2408865995311,5cyclictest131rcu_preempt11:18:512
2408865995311,14cyclictest30-21ksoftirqd/211:40:002
240886499536,7cyclictest23-21ksoftirqd/112:34:551
2408864995314,10cyclictest23-21ksoftirqd/111:35:161
2408865995244,5cyclictest2400427-21kworker/2:209:45:482
2408865995244,5cyclictest2400427-21kworker/2:209:36:182
2408865995241,8cyclictest291rcuc/210:00:102
2408865995223,10cyclictest30-21ksoftirqd/211:50:022
2408865995214,10cyclictest30-21ksoftirqd/210:24:592
240886599521,13cyclictest30-21ksoftirqd/208:10:012
240886499525,13cyclictest23-21ksoftirqd/110:30:381
2408863995228,17cyclictest275-21dbus-daemon07:50:000
240886699512,38cyclictest2411881-21kworker/u8:208:30:023
240886599517,9cyclictest30-21ksoftirqd/211:10:132
2408865995143,5cyclictest2429133-21kworker/2:011:01:002
2408865995143,5cyclictest2429133-21kworker/2:011:00:592
2408865995143,5cyclictest2400427-21kworker/2:211:26:052
2408865995143,5cyclictest2393266-21kworker/2:307:50:042
240886599514,10cyclictest30-21ksoftirqd/209:05:012
2408865995140,7cyclictest2429133-21kworker/2:010:45:232
2408865995110,7cyclictest131rcu_preempt09:54:482
2408865995110,7cyclictest131rcu_preempt09:54:482
240886499518,7cyclictest23-21ksoftirqd/108:50:021
240886499517,10cyclictest23-21ksoftirqd/111:41:031
240886499516,7cyclictest23-21ksoftirqd/110:19:451
240886499516,11cyclictest23-21ksoftirqd/110:00:411
2408864995143,5cyclictest23-21ksoftirqd/110:09:081
2408864995143,5cyclictest23-21ksoftirqd/110:09:071
240886499513,13cyclictest23-21ksoftirqd/107:55:001
240886499513,11cyclictest23-21ksoftirqd/110:00:061
240886499513,11cyclictest23-21ksoftirqd/110:00:051
240886499511,9cyclictest23-21ksoftirqd/109:16:181
2408866995034,11cyclictest1-21systemd12:40:013
240886599508,5cyclictest131rcu_preempt08:50:012
2408865995042,5cyclictest2429133-21kworker/2:011:42:042
2408865995042,5cyclictest2429133-21kworker/2:009:43:012
2408865995042,5cyclictest2393266-21kworker/2:308:26:192
2408865995041,6cyclictest2400427-21kworker/2:208:15:362
2408865995016,4cyclictest2410355-21/usr/sbin/munin07:40:002
240886499505,7cyclictest23-21ksoftirqd/111:29:351
240886499504,14cyclictest23-21ksoftirqd/108:15:021
240886499504,12cyclictest23-21ksoftirqd/110:41:201
240886499504,12cyclictest23-21ksoftirqd/110:41:201
2408864995036,11cyclictest23-21ksoftirqd/110:25:021
240886499503,10cyclictest23-21ksoftirqd/109:20:411
240886499502,13cyclictest23-21ksoftirqd/107:25:011
240886599497,11cyclictest30-21ksoftirqd/207:35:032
240886599497,11cyclictest30-21ksoftirqd/207:35:022
240886599495,7cyclictest30-21ksoftirqd/208:44:592
2408865994941,5cyclictest2429133-21kworker/2:010:41:432
2408865994941,5cyclictest2429133-21kworker/2:010:41:422
2408865994941,5cyclictest2400427-21kworker/2:207:27:052
2408865994941,5cyclictest2393266-21kworker/2:308:02:272
240886599494,10cyclictest30-21ksoftirqd/208:15:012
2408865994912,5cyclictest131rcu_preempt08:35:012
240886599491,13cyclictest30-21ksoftirqd/208:55:022
240886499498,9cyclictest23-21ksoftirqd/107:45:031
240886499498,9cyclictest23-21ksoftirqd/107:45:021
240886499493,13cyclictest23-21ksoftirqd/111:55:571
240886499491,6cyclictest0-21swapper/108:10:011
2408863994921,7cyclictest12-21ksoftirqd/007:40:010
2408863994915,11cyclictest2412642-21latency_hist08:20:020
2408866994827,14cyclictest3467602-21systemd-journal08:55:003
2408866994819,18cyclictest37-21ksoftirqd/309:20:003
2408865994840,5cyclictest2393266-21kworker/2:308:23:572
2408865994840,5cyclictest2393266-21kworker/2:308:23:562
2408865994840,5cyclictest2393266-21kworker/2:307:46:362
2408865994840,5cyclictest2393266-21kworker/2:307:11:102
2408865994839,6cyclictest2400427-21kworker/2:207:40:142
2408865994839,6cyclictest2400427-21kworker/2:207:40:132
2408865994839,5cyclictest2393266-21kworker/2:308:56:402
2408865994838,6cyclictest2393266-21kworker/2:308:38:202
2408865994838,6cyclictest2393266-21kworker/2:308:38:202
240886499486,7cyclictest23-21ksoftirqd/107:15:021
240886499485,8cyclictest23-21ksoftirqd/112:01:421
2408863994833,12cyclictest1-21systemd09:00:010
2408865994739,5cyclictest2393266-21kworker/2:309:05:052
2408865994739,5cyclictest2393266-21kworker/2:307:55:092
2408865994739,5cyclictest2393266-21kworker/2:307:15:542
2408865994739,5cyclictest2393266-21kworker/2:307:15:532
2408866994633,10cyclictest3467602-21systemd-journal10:35:013
2408866994620,21cyclictest37-21ksoftirqd/312:05:013
240886499466,7cyclictest23-21ksoftirqd/108:20:011
2408864994632,10cyclictest275-21dbus-daemon08:00:011
2408864994631,10cyclictest3467602-21systemd-journal07:20:011
2408864994631,10cyclictest3467602-21systemd-journal07:20:001
240886499461,8cyclictest1-21systemd08:25:021
240886499461,8cyclictest1-21systemd08:25:011
2408866994533,8cyclictest2497337-21sshd12:05:393
2408866994529,12cyclictest0-21swapper/310:20:033
2408864994530,11cyclictest3467602-21systemd-journal09:00:011
240886399451,4cyclictest0-21swapper/008:53:270
240886399451,29cyclictest0-21swapper/008:26:160
240886699445,6cyclictest131rcu_preempt12:15:003
2408866994437,4cyclictest2510985-21latency_hist12:35:023
2408866994429,11cyclictest3467602-21systemd-journal11:50:023
2408866994426,14cyclictest3467602-21systemd-journal07:55:003
240886699441,5cyclictest131rcu_preempt12:25:023
240886499447,7cyclictest23-21ksoftirqd/107:35:031
240886499447,7cyclictest23-21ksoftirqd/107:35:021
240886499446,8cyclictest23-21ksoftirqd/108:55:011
240886499441,5cyclictest0-21swapper/108:40:031
240886699434,5cyclictest131rcu_preempt11:25:013
2408866994329,10cyclictest3467602-21systemd-journal11:30:013
2408866994328,12cyclictest37-21ksoftirqd/310:55:553
2408866994328,12cyclictest37-21ksoftirqd/309:50:003
2408866994328,12cyclictest37-21ksoftirqd/307:30:003
2408866994322,5cyclictest37-21ksoftirqd/309:55:023
2408866994322,5cyclictest37-21ksoftirqd/309:55:023
240886499436,9cyclictest23-21ksoftirqd/107:50:011
240886399432,26cyclictest2411904-21kworker/u8:308:25:020
240886399432,26cyclictest2411904-21kworker/u8:308:25:020
240886699426,6cyclictest131rcu_preempt09:45:023
2408866994231,8cyclictest37-21ksoftirqd/310:38:173
2408866994227,12cyclictest275-21dbus-daemon10:45:013
2408866994227,12cyclictest275-21dbus-daemon10:45:003
2408866994227,11cyclictest3467602-21systemd-journal09:11:023
2408866994222,14cyclictest37-21ksoftirqd/309:35:043
2408866994217,5cyclictest37-21ksoftirqd/311:51:443
2408865994210,8cyclictest30-21ksoftirqd/207:10:032
2408865994210,8cyclictest30-21ksoftirqd/207:10:022
240886499422,11cyclictest23-21ksoftirqd/108:29:191
240886699416,6cyclictest131rcu_preempt11:44:133
2408866994125,12cyclictest275-21dbus-daemon12:20:013
2408866994125,12cyclictest275-21dbus-daemon12:20:003
2408866994117,16cyclictest37-21ksoftirqd/312:30:023
2408866994117,16cyclictest37-21ksoftirqd/312:30:013
240886699411,5cyclictest0-21swapper/311:32:133
240886699411,13cyclictest0-21swapper/309:31:103
240886499413,11cyclictest23-21ksoftirqd/108:00:391
2408863994126,11cyclictest12-21ksoftirqd/007:45:020
2408863994126,11cyclictest12-21ksoftirqd/007:45:010
240886399411,35cyclictest3467602-21systemd-journal07:28:560
2408866994027,10cyclictest37-21ksoftirqd/308:50:023
2408866994026,11cyclictest37-21ksoftirqd/310:23:463
240886699402,5cyclictest131rcu_preempt08:15:023
2408866994022,16cyclictest37-21ksoftirqd/310:04:593
2408866994014,6cyclictest131rcu_preempt10:05:303
2408866994014,6cyclictest131rcu_preempt10:05:293
2408866994012,15cyclictest2464070-21grep10:55:023
2408866994012,15cyclictest2464070-21grep10:55:013
2408863994033,4cyclictest16950irq/224-800000007:59:450
2408863994030,6cyclictest2411934-21grep08:10:020
240886399402,5cyclictest2409606-21munin-run07:25:000
240886699396,5cyclictest2469445-21ssh11:06:063
240886699394,4cyclictest131rcu_preempt10:47:173
2408866993926,9cyclictest1-21systemd09:00:013
2408866993925,11cyclictest2412531-21cat08:20:013
2408866993925,10cyclictest37-21ksoftirqd/309:08:503
2408866993923,12cyclictest3467602-21systemd-journal11:00:033
2408866993923,12cyclictest3467602-21systemd-journal11:00:033
240886499391,5cyclictest0-21swapper/108:30:181
240886399395,4cyclictest131rcu_preempt07:35:020
240886399395,4cyclictest131rcu_preempt07:35:010
2408866993826,9cyclictest2409793-21sed07:25:033
2408866993825,10cyclictest2443806-21ssh10:11:233
2408866993824,11cyclictest37-21ksoftirqd/309:20:233
2408866993824,11cyclictest37-21ksoftirqd/308:35:013
2408866993824,10cyclictest2410352-21systemctl07:39:593
2408866993822,5cyclictest131rcu_preempt11:36:123
2408866993815,6cyclictest131rcu_preempt09:26:203
2408863993827,8cyclictest12-21ksoftirqd/007:55:020
240886399381,32cyclictest2408900-21latency_hist07:10:020
240886399381,32cyclictest2408900-21latency_hist07:10:020
2408866993725,9cyclictest37-21ksoftirqd/308:42:403
2408866993725,9cyclictest37-21ksoftirqd/308:20:363
2408866993725,9cyclictest37-21ksoftirqd/308:20:353
2408866993724,9cyclictest37-21ksoftirqd/307:44:533
2408866993724,9cyclictest37-21ksoftirqd/307:44:523
2408866993718,8cyclictest131rcu_preempt11:11:533
2408866993718,7cyclictest131rcu_preempt10:27:123
2408866993714,13cyclictest37-21ksoftirqd/307:19:343
2408866993714,13cyclictest37-21ksoftirqd/307:19:343
2408866993617,3cyclictest37-21ksoftirqd/307:31:393
2408866993617,3cyclictest37-21ksoftirqd/307:31:383
2408863993627,5cyclictest2413174-21sshd08:30:120
2408863993624,7cyclictest2413743-21grep08:44:590
240886699351,8cyclictest0-21swapper/309:03:383
240886699351,6cyclictest131rcu_preempt07:49:593
2408863993522,10cyclictest12-21ksoftirqd/009:02:450
2408863993517,14cyclictest2413966-21sshd08:48:390
240886399351,31cyclictest0-21swapper/008:00:030
240886699349,5cyclictest131rcu_preempt07:10:113
2408866993421,11cyclictest37-21ksoftirqd/308:10:003
240886499345,6cyclictest23-21ksoftirqd/107:10:031
240886499345,6cyclictest23-21ksoftirqd/107:10:021
2408866993322,9cyclictest37-21ksoftirqd/308:00:003
2408866992910,11cyclictest131rcu_preempt07:10:033
2408866992910,11cyclictest131rcu_preempt07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional