You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-06 - 05:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Thu Aug 06, 2026 00:45:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2339621999233,55cyclictest2388161-21ssh22:38:260
233962199862,13cyclictest48-21kcompactd000:00:120
2339621998537,30cyclictest16850irq/134-800000021:50:010
233962199803,71cyclictest2373547-21ssh22:07:520
2339621997630,30cyclictest2438590-21ssh00:23:270
2339621997615,56cyclictest16850irq/134-800000000:05:520
2339621997615,56cyclictest16850irq/134-800000000:05:520
233962199755,5cyclictest2444544-21ssh00:35:380
233962199752,5cyclictest2401024-21latency_hist23:05:040
233962199752,5cyclictest2339811-21munin-run19:15:010
233962199751,70cyclictest48-21kcompactd023:30:070
2339628997410,50cyclictest2386610-21ssh22:35:022
2339628997410,50cyclictest2386610-21ssh22:35:022
2339621997428,28cyclictest2397312-21ssh22:57:260
2339621997426,43cyclictest2434984-21ssh00:15:450
2339621997426,43cyclictest2434984-21ssh00:15:440
233962199741,68cyclictest2356699-21ssh21:32:330
233962199737,15cyclictest2424636-21ssh23:54:330
2339621997327,29cyclictest2370126-21ssh22:00:240
2339621997320,49cyclictest16850irq/134-800000021:54:430
233962199731,7cyclictest2420881-21ssh23:46:260
2339621997218,50cyclictest2440199-21ssh00:26:400
2339621997218,50cyclictest2440199-21ssh00:26:390
2339621997214,52cyclictest16850irq/134-800000021:29:060
2339621997211,56cyclictest16850irq/134-800000023:09:420
2339621997167,2cyclictest0-21swapper/021:45:020
233962199712,10cyclictest2408176-21latency_hist23:20:020
233962199711,8cyclictest2347818-21ssh21:14:090
2339621997117,49cyclictest16850irq/134-800000022:46:060
233962199711,6cyclictest2345001-21munin-run20:55:010
233962199711,6cyclictest2345001-21munin-run20:55:010
2339621997115,52cyclictest16850irq/134-800000022:44:340
2339621997115,52cyclictest16850irq/134-800000022:44:330
2339621997026,27cyclictest2352493-21ssh21:23:560
2339621997012,53cyclictest16850irq/134-800000022:13:510
2339621997010,55cyclictest16850irq/134-800000023:57:060
2339621997010,55cyclictest16850irq/134-800000023:57:050
2339621996926,26cyclictest2383739-21ssh22:29:240
233962199692,5cyclictest2349829-21ssh21:18:180
2339621996925,26cyclictest2432816-21ssh00:11:150
2339621996923,42cyclictest2344720-21sshd20:46:260
233962199691,7cyclictest1-21systemd23:11:260
2339621996917,47cyclictest2340026-21sshd19:15:040
2339621996914,49cyclictest16850irq/134-800000022:33:520
2339621996914,49cyclictest16850irq/134-800000022:33:510
2339621996913,51cyclictest16850irq/134-800000022:20:140
2339621996913,51cyclictest16850irq/134-800000022:20:140
233962199691,13cyclictest2358965-21ssh21:37:180
2339621996827,24cyclictest2369311-21ssh21:59:130
2339621996825,26cyclictest2395148-21ssh22:52:530
233963199672,17cyclictest2427083-21ssh23:59:403
233963199672,17cyclictest2427083-21ssh23:59:403
2339631996623,39cyclictest2340818-21sshd19:30:203
2339631996616,45cyclictest2403827-1kworker/u9:023:38:553
233962199662,6cyclictest2342921-21munin-run20:15:000
2339621996626,23cyclictest2419441-21ssh23:43:370
2339621996625,24cyclictest2443255-21ssh00:33:080
2339621996625,24cyclictest2443255-21ssh00:33:080
2339621996613,47cyclictest16850irq/134-800000022:16:060
2339621996529,32cyclictest2408941-21rm23:21:230
2339621996529,32cyclictest2408941-21rm23:21:230
233962199651,4cyclictest3467602-21systemd-journal21:01:310
2339631996424,36cyclictest2413242-21sh23:30:233
2339631996416,43cyclictest2342907-21cron20:15:003
2339628996447,9cyclictest2329134-21kworker/2:020:30:002
2339621996426,34cyclictest2410847-21ssh23:25:160
2339621996324,35cyclictest2415908-21ssh23:35:560
2339631996224,34cyclictest2341125-21tr19:40:003
233963199622,16cyclictest2344256-21munin-run20:40:013
233963199622,16cyclictest2344256-21munin-run20:40:013
2339631996215,7cyclictest2398412-21modprobe23:00:003
2339631996215,42cyclictest2230356-1kworker/u9:221:13:523
2339628996234,5cyclictest2415262-21kworker/2:100:08:572
2339628996234,5cyclictest2415262-21kworker/2:100:08:562
2339621996222,34cyclictest2343168-21cron20:20:000
233962199621,8cyclictest2342154-21tr20:00:010
2339631996120,37cyclictest2390560-21sh22:43:203
2339631996120,37cyclictest2390560-21sh22:43:203
2339631996114,42cyclictest2405867-21cat23:15:023
2339628996149,6cyclictest2329134-21kworker/2:019:55:002
233963199604,17cyclictest2434507-21modprobe00:15:003
2339631996023,32cyclictest2352415-21sh21:23:473
233963199602,16cyclictest2383841-21ssh22:29:373
2339621996024,31cyclictest1-21systemd19:50:000
233962199602,19cyclictest2341144-21munin-run19:40:010
2339631995930,26cyclictest37-21ksoftirqd/323:02:543
2339631995912,42cyclictest2376830-21cron22:15:003
233963199591,17cyclictest2345519-21cron21:05:003
2339628995951,5cyclictest2344711-21kworker/2:221:45:022
233962199593,6cyclictest364-21snmpd20:25:340
233962199591,6cyclictest2344195-21sshd20:38:260
233962199591,6cyclictest2344195-21sshd20:38:260
233963199583,9cyclictest2350578-21cron21:20:003
233963199582,7cyclictest2393704-21unin-run22:50:003
2339631995824,29cyclictest2445538-21sh00:37:533
233963199582,17cyclictest2412942-21ssh23:30:013
233963199582,14cyclictest2409897-21rm23:23:343
233963199582,14cyclictest2409897-21rm23:23:343
2339628995849,5cyclictest2344711-21kworker/2:223:18:062
233962199581,15cyclictest2343443-21sshd20:24:260
233962199581,15cyclictest2343443-21sshd20:24:260
233963199572,9cyclictest275-21dbus-daemon19:25:013
233963199572,9cyclictest2439290-21ssh00:25:003
2339631995712,41cyclictest275-21dbus-daemon23:20:003
233963199571,14cyclictest48-21kcompactd022:05:283
233963199571,10cyclictest2356859-21ssh21:33:043
2339631995710,40cyclictest2367285-21modprobe21:55:003
2339628995748,6cyclictest2415262-21kworker/2:123:40:182
233963199562,12cyclictest2440594-21ssh00:27:403
233963199562,12cyclictest2440594-21ssh00:27:403
233963199562,12cyclictest2378727-21ssh22:18:493
233963199562,10cyclictest364-21snmpd23:08:203
2339631995616,35cyclictest2343204-21basename20:20:003
233963199561,19cyclictest2381619-21munin-plugin-st22:25:013
233963199561,19cyclictest2381619-21munin-plugin-st22:25:003
233963199561,14cyclictest2418066-21ssh23:40:253
233963199561,11cyclictest2370330-21ssh22:00:563
2339628995646,6cyclictest2344711-21kworker/2:221:36:302
2339628995638,10cyclictest309-21in:imuxsock20:20:012
2339628995627,7cyclictest2344711-21kworker/2:221:49:492
2339621995649,4cyclictest16950irq/224-800000019:35:010
2339621995627,25cyclictest3467602-21systemd-journal20:30:020
233963199552,6cyclictest2429770-21sshd00:05:013
2339631995522,29cyclictest2384605-21sh22:30:503
2339631995522,29cyclictest2384605-21sh22:30:503
233963199552,14cyclictest2435832-21ssh00:17:383
233963199552,14cyclictest2435832-21ssh00:17:373
233963199552,14cyclictest2424740-21ssh23:54:473
233963199552,12cyclictest2340033-21sshd19:16:193
233963199551,14cyclictest2430041-21sh00:05:213
233963199551,14cyclictest2430041-21sh00:05:203
233963199551,14cyclictest2368085-21ssh21:56:223
233963199551,14cyclictest2354042-21ssh21:27:033
233963199551,13cyclictest2359420-21ssh21:38:243
2339628995547,5cyclictest2415262-21kworker/2:100:28:172
2339628995547,5cyclictest2415262-21kworker/2:100:28:172
2339628995545,5cyclictest2364937-21kworker/2:322:16:042
2339628995539,8cyclictest309-21in:imuxsock21:55:012
2339628995535,16cyclictest2388841-21cron22:40:002
2339626995537,13cyclictest3467602-21systemd-journal23:45:001
233962199552,13cyclictest2342385-21sshd20:00:190
2339631995422,29cyclictest3467562-21sshd20:50:223
2339631995422,29cyclictest3467562-21sshd20:50:223
233963199542,15cyclictest2386976-21ssh22:35:423
233963199542,14cyclictest2343461-21munin-run20:25:013
233963199542,14cyclictest2343461-21munin-run20:25:013
2339631995414,34cyclictest0-21swapper/300:30:313
2339631995414,34cyclictest0-21swapper/300:30:303
2339628995446,5cyclictest2415262-21kworker/2:123:39:542
2339628995446,5cyclictest2415262-21kworker/2:100:32:392
2339628995446,5cyclictest2415262-21kworker/2:100:32:382
2339628995446,5cyclictest2364937-21kworker/2:322:07:132
2339628995446,5cyclictest2329134-21kworker/2:020:25:022
2339628995446,5cyclictest2329134-21kworker/2:020:25:022
2339628995445,5cyclictest2415262-21kworker/2:100:04:252
2339628995444,6cyclictest2364937-21kworker/2:322:11:532
2339628995444,6cyclictest2344711-21kworker/2:222:50:522
233962199542,6cyclictest2340600-21munin-run19:30:010
233962199542,5cyclictest2345756-21sshd21:07:260
2339621995418,32cyclictest2339551-21latency_hist19:10:020
2339621995418,32cyclictest2339551-21latency_hist19:10:010
233962199541,14cyclictest364-21snmpd19:20:170
233963199532,8cyclictest2395496-21ssh22:53:453
233963199531,13cyclictest2361456-21sh21:42:323
2339628995345,5cyclictest2344711-21kworker/2:221:57:102
2339628995345,5cyclictest2344711-21kworker/2:221:31:402
2339628995345,5cyclictest2344711-21kworker/2:221:27:042
2339628995344,5cyclictest2344711-21kworker/2:223:24:252
2339628995344,5cyclictest2344711-21kworker/2:223:24:242
2339628995323,9cyclictest2364937-21kworker/2:322:22:382
2339628995323,9cyclictest2364937-21kworker/2:322:22:372
233962199532,6cyclictest364-21snmpd20:05:060
233962199532,4cyclictest2345229-21sshd20:58:260
233962199531,6cyclictest2344467-21sshd20:41:260
233962199531,6cyclictest2341885-21munin-run19:55:000
233963199523,14cyclictest2339823-21munin-run19:15:013
233963199522,17cyclictest2342667-21munin-run20:10:003
233963199521,14cyclictest1-21systemd20:00:013
233963199521,12cyclictest2420781-21ssh23:46:133
233963199521,12cyclictest2340610-21sh19:30:013
233963199521,10cyclictest2363796-21ssh21:47:243
2339628995244,5cyclictest2364937-21kworker/2:322:47:412
2339628995243,6cyclictest2364937-21kworker/2:322:40:302
2339628995243,6cyclictest2364937-21kworker/2:322:40:302
2339628995242,6cyclictest2344711-21kworker/2:223:58:292
2339628995242,6cyclictest2344711-21kworker/2:223:58:292
2339628995242,6cyclictest2329134-21kworker/2:020:35:002
2339631995121,26cyclictest37-21ksoftirqd/319:45:023
2339628995143,5cyclictest2439327-21kworker/2:000:39:152
2339628995143,5cyclictest2344711-21kworker/2:223:46:122
2339628995143,5cyclictest2344711-21kworker/2:223:26:272
2339628995143,5cyclictest2344711-21kworker/2:222:59:312
2339628995143,5cyclictest2344711-21kworker/2:200:16:352
2339628995143,5cyclictest2344711-21kworker/2:200:16:352
2339628995143,5cyclictest2344711-21kworker/2:200:13:062
2339628995143,5cyclictest2329134-21kworker/2:021:17:252
2339628995143,5cyclictest2329134-21kworker/2:020:45:022
2339628995142,6cyclictest2364937-21kworker/2:322:26:502
2339628995142,6cyclictest2344711-21kworker/2:223:50:442
2339628995141,6cyclictest2364937-21kworker/2:323:11:072
2339628995141,6cyclictest2329134-21kworker/2:021:10:452
2339626995132,13cyclictest3467602-21systemd-journal19:50:001
233963199502,15cyclictest2343965-21munin-run20:35:003
2339628995042,5cyclictest2329134-21kworker/2:021:21:452
2339628995041,5cyclictest2329134-21kworker/2:021:10:022
2339628995040,6cyclictest2344711-21kworker/2:223:02:452
233962699504,3cyclictest131rcu_preempt00:00:021
233962699504,3cyclictest131rcu_preempt00:00:021
233963199495,10cyclictest2345785-21latency_hist21:10:003
233963199494,10cyclictest2341628-21sh19:50:003
233963199492,8cyclictest275-21dbus-daemon20:50:013
233963199491,12cyclictest2345215-21sshd20:55:253
2339628994941,5cyclictest2344711-21kworker/2:223:35:002
2339628994940,6cyclictest2329134-21kworker/2:019:35:012
2339628994940,5cyclictest2415262-21kworker/2:100:23:522
2339628994928,16cyclictest3467602-21systemd-journal20:15:002
233963199481,10cyclictest364-21snmpd19:50:033
2339628994840,5cyclictest2364937-21kworker/2:322:03:402
2339628994839,6cyclictest2364937-21kworker/2:323:06:022
233962699484,3cyclictest131rcu_preempt23:02:181
2339626994832,12cyclictest1-21systemd21:05:011
2339626994831,13cyclictest1-21systemd23:30:001
2339626994829,14cyclictest1-21systemd21:50:011
233962199482,8cyclictest2341370-21cron19:45:000
233963199462,12cyclictest364-21snmpd20:02:093
2339628994638,5cyclictest2329134-21kworker/2:019:25:172
2339628994638,5cyclictest2329134-21kworker/2:019:18:252
2339628994636,6cyclictest2329134-21kworker/2:020:54:272
2339628994636,6cyclictest2329134-21kworker/2:020:54:272
2339628994636,5cyclictest2329134-21kworker/2:020:10:022
2339626994632,11cyclictest23-21ksoftirqd/100:07:491
2339626994632,11cyclictest23-21ksoftirqd/100:07:491
2339626994624,13cyclictest2358106-21ssh21:35:241
233963199456,5cyclictest0-21swapper/319:10:023
233963199456,5cyclictest0-21swapper/319:10:023
2339628994537,5cyclictest2329134-21kworker/2:019:45:492
233962699458,33cyclictest23-21ksoftirqd/120:05:031
233962699454,3cyclictest131rcu_preempt21:34:091
233962699452,3cyclictest131rcu_preempt22:05:001
233962699451,3cyclictest131rcu_preempt21:12:281
233962699451,3cyclictest131rcu_preempt00:22:301
2339631994410,4cyclictest0-21swapper/320:45:003
2339628994437,4cyclictest2329134-21kworker/2:020:56:192
2339628994436,5cyclictest2329134-21kworker/2:019:25:002
2339626994435,5cyclictest2417615-21sendmail23:39:591
2339626994432,10cyclictest23-21ksoftirqd/123:45:081
233962699441,3cyclictest131rcu_preempt22:12:161
233962699441,3cyclictest131rcu_preempt21:53:421
233963199431,12cyclictest2343684-21sshd20:26:263
233962699433,3cyclictest131rcu_preempt22:17:381
2339626994329,11cyclictest23-21ksoftirqd/121:45:021
2339626994328,12cyclictest23-21ksoftirqd/100:15:001
2339626994328,11cyclictest2429778-21wc00:05:021
233962699432,3cyclictest131rcu_preempt22:45:171
2339626994311,6cyclictest131rcu_preempt21:25:021
2339628994234,5cyclictest2329134-21kworker/2:019:55:462
2339628994233,5cyclictest2329134-21kworker/2:021:04:382
2339628994233,5cyclictest2329134-21kworker/2:020:05:012
2339626994229,10cyclictest275-21dbus-daemon19:25:011
2339626994228,10cyclictest23-21ksoftirqd/100:38:441
2339626994227,11cyclictest2389824-21ssh22:41:401
2339626994227,11cyclictest2389824-21ssh22:41:401
2339628994111,6cyclictest131rcu_preempt20:48:262
2339626994132,6cyclictest2345784-21diskmemload22:00:021
2339626994128,9cyclictest23-21ksoftirqd/123:54:001
2339626994128,10cyclictest2343455-21cron20:25:001
2339626994128,10cyclictest2343455-21cron20:25:001
2339626994127,11cyclictest23-21ksoftirqd/122:22:561
2339626994127,11cyclictest23-21ksoftirqd/122:22:561
2339626994126,11cyclictest23-21ksoftirqd/121:18:351
2339626994126,11cyclictest23-21ksoftirqd/100:16:371
2339626994126,11cyclictest23-21ksoftirqd/100:16:361
2339626994124,11cyclictest309-21in:imuxsock22:30:001
2339626994122,13cyclictest275-21dbus-daemon22:35:011
2339626994122,13cyclictest275-21dbus-daemon22:35:011
2339626994121,14cyclictest3467602-21systemd-journal19:30:001
233962699411,3cyclictest131rcu_preempt22:37:241
2339628994031,6cyclictest2329134-21kworker/2:019:11:012
2339626994030,8cyclictest23-21ksoftirqd/123:21:481
2339626994030,8cyclictest23-21ksoftirqd/123:21:471
233962699402,29cyclictest2343171-21kworker/u8:020:20:011
2339628993927,8cyclictest309-21in:imuxsock20:40:002
2339628993927,8cyclictest309-21in:imuxsock20:40:002
233962699394,6cyclictest2342159-21sendmail20:00:011
2339626993930,7cyclictest23-21ksoftirqd/122:09:291
2339626993928,9cyclictest23-21ksoftirqd/121:29:041
2339626993928,8cyclictest23-21ksoftirqd/100:26:011
2339626993928,8cyclictest23-21ksoftirqd/100:26:001
2339626993927,9cyclictest23-21ksoftirqd/119:40:001
2339626993926,10cyclictest23-21ksoftirqd/122:57:361
2339626993925,9cyclictest309-21in:imuxsock20:51:261
2339626993925,9cyclictest309-21in:imuxsock20:51:261
2339626993925,10cyclictest2345255-21cat21:00:001
2339626993925,10cyclictest23-21ksoftirqd/100:32:471
2339626993925,10cyclictest23-21ksoftirqd/100:32:471
2339626993924,11cyclictest23-21ksoftirqd/123:05:381
2339626993923,12cyclictest23-21ksoftirqd/123:15:021
2339628993830,5cyclictest2329134-21kworker/2:019:36:152
2339628993824,11cyclictest30-21ksoftirqd/219:40:432
2339626993825,10cyclictest23-21ksoftirqd/119:55:001
2339626993824,10cyclictest23-21ksoftirqd/123:32:341
2339626993730,4cyclictest2340072-21latency_hist19:20:011
2339626993728,6cyclictest23-21ksoftirqd/119:10:021
2339626993728,6cyclictest23-21ksoftirqd/119:10:021
2339626993727,8cyclictest23-21ksoftirqd/120:30:001
2339626993727,7cyclictest23-21ksoftirqd/122:52:121
2339626993726,8cyclictest23-21ksoftirqd/119:35:031
2339626993725,8cyclictest23-21ksoftirqd/120:40:011
2339626993725,8cyclictest23-21ksoftirqd/120:40:011
2339626993724,10cyclictest23-21ksoftirqd/123:16:041
2339626993722,12cyclictest23-21ksoftirqd/120:35:011
2339626993625,8cyclictest23-21ksoftirqd/120:05:011
2339626993623,9cyclictest23-21ksoftirqd/120:50:001
2339626993523,8cyclictest23-21ksoftirqd/120:45:001
2339626993523,6cyclictest309-21in:imuxsock20:15:001
2339626993519,11cyclictest275-21dbus-daemon19:15:011
2339626993416,13cyclictest275-21dbus-daemon19:45:011
2339628993320,9cyclictest2339736-21cat19:10:022
2339628993320,9cyclictest2339736-21cat19:10:022
2339626993319,10cyclictest2345873-21cut21:10:031
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional