You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-14 - 12:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Tue Apr 14, 2026 00:45:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3508518998025,38cyclictest3563652-21ssh23:03:420
3508518998024,52cyclictest3524344-21ssh21:32:160
3508518998024,52cyclictest3524344-21ssh21:32:160
3508518997928,33cyclictest3546568-21ssh22:24:070
350851899772,71cyclictest0-21swapper/021:25:010
350851899771,12cyclictest3588403-21sshd00:00:560
3508518997628,44cyclictest3517800-21ssh21:17:040
3508518997329,29cyclictest3576813-21ssh23:34:290
3508518997314,40cyclictest16850irq/134-800000022:31:440
3508518997312,41cyclictest16850irq/134-800000023:23:470
3508518997228,40cyclictest3525677-21ssh21:35:100
3508518997129,25cyclictest3552745-21ssh22:38:260
350851899712,6cyclictest3536292-21cat22:00:020
350851899712,6cyclictest3536292-21cat22:00:010
3508518997119,48cyclictest3533987-21ssh21:54:570
350851899701,63cyclictest48-21kcompactd019:35:200
350851899701,63cyclictest48-21kcompactd019:35:190
350851899697,12cyclictest3574035-21ssh23:27:480
3508518996930,35cyclictest3554550-21ssh22:42:310
350851899692,8cyclictest3567798-21ssh23:13:220
350851899692,8cyclictest3567798-21ssh23:13:220
3508518996829,35cyclictest3528642-21ssh21:42:100
3508518996829,35cyclictest3528642-21ssh21:42:100
3508518996825,26cyclictest3594015-21ssh00:14:250
3508518996819,32cyclictest3515073-21ssh21:10:360
3508518996743,20cyclictest16950irq/224-800000022:57:080
350851899672,12cyclictest3597749-21sshd00:22:570
3508518996718,45cyclictest0-21swapper/022:26:120
350851899663,17cyclictest3565391-21ssh23:07:370
3508518996629,35cyclictest0-21swapper/022:17:150
3508518996629,35cyclictest0-21swapper/022:17:140
3508518996616,45cyclictest16850irq/134-800000022:12:130
3508518996613,49cyclictest3570123-21ssh23:18:500
350851899658,52cyclictest16850irq/134-800000000:15:020
350851899656,14cyclictest3586648-21ssh23:57:070
350851899656,14cyclictest3586648-21ssh23:57:070
350851899652,17cyclictest3513911-21munin-run20:54:590
3508518996519,29cyclictest0-21swapper/000:05:180
3508518996518,31cyclictest3531596-21ssh21:49:200
3508518996423,37cyclictest3556071-21sshd22:45:570
3508518996422,38cyclictest0-21swapper/019:20:010
3508518996417,42cyclictest16850irq/134-800000000:30:520
3508518996413,46cyclictest3578269-21ssh23:37:360
3508518996330,30cyclictest3523364-21latency_hist21:30:010
3508518996327,29cyclictest275-21dbus-daemon23:45:000
350851899631,18cyclictest3605061-21cat00:40:020
350851899631,14cyclictest3512845-21sshd20:32:110
3508532996238,15cyclictest37-21ksoftirqd/321:33:303
3508532996238,15cyclictest37-21ksoftirqd/321:33:303
3508518996230,28cyclictest3582119-21ssh23:46:270
350851899622,14cyclictest3512369-21munin-run20:25:000
3508518996220,38cyclictest3584166-21ssh23:51:160
350851899621,12cyclictest3513134-21munin-run20:40:000
350851899612,14cyclictest3509256-21munin-run19:25:010
3508518996117,15cyclictest275-21dbus-daemon21:10:010
3508532996027,10cyclictest37-21ksoftirqd/323:39:463
350851899608,3cyclictest0-21swapper/019:46:590
350851899603,15cyclictest3559768-21cron22:55:000
350851899602,52cyclictest0-21swapper/021:00:000
350851899602,15cyclictest3510289-21munin-run19:45:000
350853299599,8cyclictest37-21ksoftirqd/300:00:033
350853299599,8cyclictest37-21ksoftirqd/300:00:033
3508524995917,9cyclictest23-21ksoftirqd/121:49:161
3508518995929,25cyclictest12-21ksoftirqd/000:25:480
350853299587,8cyclictest37-21ksoftirqd/323:20:023
3508528995848,6cyclictest3542623-21kworker/2:223:22:332
350852499588,9cyclictest23-21ksoftirqd/100:04:561
3508518995829,24cyclictest12-21ksoftirqd/022:07:140
350851899581,9cyclictest3511797-21sshd20:11:570
350853299577,9cyclictest37-21ksoftirqd/322:42:013
3508532995739,13cyclictest275-21dbus-daemon19:40:013
3508532995739,13cyclictest275-21dbus-daemon19:40:013
3508532995727,11cyclictest37-21ksoftirqd/323:02:513
3508532995713,8cyclictest37-21ksoftirqd/300:38:363
3508528995748,6cyclictest3501986-21kworker/2:021:15:012
3508528995746,8cyclictest3501986-21kworker/2:023:25:072
3508528995746,6cyclictest3523700-21kworker/2:122:00:022
3508528995746,6cyclictest3523700-21kworker/2:122:00:022
350851899572,6cyclictest3512635-21munin-run20:30:010
350851899571,13cyclictest364-21snmpd20:05:230
350851899571,13cyclictest364-21snmpd20:05:220
350851899571,11cyclictest3511302-21sshd20:03:560
3508532995621,10cyclictest37-21ksoftirqd/322:55:173
3508528995646,5cyclictest3523700-21kworker/2:122:05:012
350852499569,8cyclictest23-21ksoftirqd/120:35:021
3508524995647,6cyclictest221rcuc/100:35:001
350852499563,9cyclictest23-21ksoftirqd/123:37:371
350851899562,6cyclictest3509763-21true19:35:010
350851899562,14cyclictest3536794-21ssh22:00:580
350851899562,14cyclictest3509500-21modprobe19:30:000
350851899561,8cyclictest3508698-21sshd19:14:580
350851899561,8cyclictest3508698-21sshd19:14:570
350853299556,8cyclictest37-21ksoftirqd/321:41:593
350853299556,8cyclictest37-21ksoftirqd/321:41:593
3508532995510,5cyclictest131rcu_preempt22:45:153
350852499557,11cyclictest23-21ksoftirqd/122:20:031
350852499557,11cyclictest23-21ksoftirqd/122:20:031
3508524995512,9cyclictest23-21ksoftirqd/122:40:011
350851899553,8cyclictest3508512-21cyclictest19:10:010
350851899553,17cyclictest3512120-21modprobe20:20:010
350851899553,17cyclictest3512120-21modprobe20:20:000
350851899551,9cyclictest3510765-21sshd19:52:050
350851899551,9cyclictest3510765-21sshd19:52:040
350851899551,12cyclictest364-21snmpd19:59:490
3508532995410,10cyclictest37-21ksoftirqd/300:29:183
3508528995441,7cyclictest3501986-21kworker/2:023:46:042
350852499547,12cyclictest23-21ksoftirqd/123:41:461
350852499545,10cyclictest23-21ksoftirqd/100:38:351
350852499544,16cyclictest23-21ksoftirqd/123:47:551
3508524995416,10cyclictest23-21ksoftirqd/123:26:051
3508524995415,8cyclictest23-21ksoftirqd/123:55:041
3508524995415,8cyclictest23-21ksoftirqd/123:55:031
3508518995434,14cyclictest3513653-21cron20:50:010
3508518995434,14cyclictest3513653-21cron20:50:010
350853299539,4cyclictest131rcu_preempt22:27:493
350853299539,11cyclictest37-21ksoftirqd/323:50:013
350853299537,13cyclictest37-21ksoftirqd/300:00:513
3508532995343,7cyclictest361rcuc/322:00:033
3508532995310,4cyclictest131rcu_preempt21:20:513
3508528995345,5cyclictest3501986-21kworker/2:021:20:022
3508528995343,6cyclictest3592117-21kworker/2:300:22:202
3508528995343,6cyclictest3501986-21kworker/2:021:25:132
3508528995342,6cyclictest3501986-21kworker/2:021:08:562
350852499536,7cyclictest23-21ksoftirqd/121:39:101
350852499534,13cyclictest23-21ksoftirqd/121:41:151
350852499534,13cyclictest23-21ksoftirqd/121:41:141
350853299529,11cyclictest37-21ksoftirqd/321:58:413
350853299529,11cyclictest37-21ksoftirqd/321:58:403
350853299526,9cyclictest37-21ksoftirqd/323:29:073
3508532995234,13cyclictest275-21dbus-daemon20:50:013
3508532995234,13cyclictest275-21dbus-daemon20:50:003
3508528995243,6cyclictest3542623-21kworker/2:200:04:242
3508528995243,6cyclictest3523700-21kworker/2:121:39:022
3508528995243,6cyclictest3501986-21kworker/2:020:50:032
3508528995243,6cyclictest3501986-21kworker/2:020:50:032
3508528995243,5cyclictest3598645-21kworker/2:000:29:192
3508528995243,5cyclictest3542623-21kworker/2:223:15:012
3508528995243,5cyclictest3542623-21kworker/2:223:15:012
3508528995243,5cyclictest3542623-21kworker/2:200:16:312
3508528995243,5cyclictest3501986-21kworker/2:021:31:112
3508528995243,5cyclictest3501986-21kworker/2:021:31:102
3508528995242,6cyclictest3542623-21kworker/2:222:57:372
3508528995241,8cyclictest3501986-21kworker/2:019:10:022
350852499524,12cyclictest23-21ksoftirqd/122:48:351
350852499524,10cyclictest23-21ksoftirqd/123:33:541
3508524995213,9cyclictest23-21ksoftirqd/100:25:431
350852499521,15cyclictest3547374-21ssh22:25:311
3508524995210,9cyclictest23-21ksoftirqd/121:31:451
3508524995210,9cyclictest23-21ksoftirqd/121:31:441
350851899521,8cyclictest364-21snmpd21:03:030
350851899521,14cyclictest3513406-21munin-plugin-st20:45:000
350853299519,8cyclictest37-21ksoftirqd/300:05:143
350853299518,10cyclictest37-21ksoftirqd/323:05:023
350853299517,8cyclictest37-21ksoftirqd/323:12:123
350853299517,8cyclictest37-21ksoftirqd/323:12:113
350853299517,7cyclictest37-21ksoftirqd/322:16:283
350853299517,7cyclictest37-21ksoftirqd/322:16:273
350853299517,12cyclictest37-21ksoftirqd/322:39:343
350853299516,7cyclictest37-21ksoftirqd/322:32:473
3508532995134,12cyclictest3467602-21systemd-journal20:40:003
350853299512,10cyclictest37-21ksoftirqd/321:39:033
3508532995111,9cyclictest37-21ksoftirqd/322:50:333
3508528995143,5cyclictest3501986-21kworker/2:021:24:352
3508528995143,5cyclictest3501986-21kworker/2:019:45:022
3508528995142,6cyclictest3523700-21kworker/2:122:07:532
3508528995142,5cyclictest3501986-21kworker/2:022:32:512
3508528995140,6cyclictest3523700-21kworker/2:121:50:342
3508528995139,6cyclictest3501986-21kworker/2:022:37:442
350852499519,5cyclictest131rcu_preempt23:24:231
350852499517,9cyclictest23-21ksoftirqd/123:15:011
350852499517,9cyclictest23-21ksoftirqd/123:15:011
350852499517,8cyclictest23-21ksoftirqd/123:15:201
350852499517,7cyclictest23-21ksoftirqd/123:05:011
350852499516,8cyclictest23-21ksoftirqd/122:42:171
350852499516,4cyclictest131rcu_preempt22:33:401
350852499515,9cyclictest3517100-21ssh21:15:161
350852499514,7cyclictest23-21ksoftirqd/122:55:461
3508524995113,9cyclictest23-21ksoftirqd/123:52:581
350853299509,11cyclictest37-21ksoftirqd/321:47:033
350853299506,7cyclictest37-21ksoftirqd/320:45:023
350853299506,11cyclictest37-21ksoftirqd/300:22:253
350853299506,10cyclictest37-21ksoftirqd/322:06:503
350853299505,8cyclictest37-21ksoftirqd/321:16:153
350853299505,6cyclictest37-21ksoftirqd/321:29:573
3508532995040,7cyclictest361rcuc/323:30:043
3508532995024,10cyclictest37-21ksoftirqd/323:23:323
3508528995042,5cyclictest3598645-21kworker/2:000:38:082
3508528995042,5cyclictest3542623-21kworker/2:223:59:042
3508528995042,5cyclictest3542623-21kworker/2:223:59:042
3508528995042,5cyclictest3542623-21kworker/2:223:04:572
3508528995042,5cyclictest3542623-21kworker/2:222:51:092
3508528995042,5cyclictest3542623-21kworker/2:200:11:022
3508528995042,5cyclictest3523700-21kworker/2:121:44:422
3508528995042,5cyclictest3523700-21kworker/2:121:44:412
3508528995041,6cyclictest3501986-21kworker/2:023:39:582
3508528995040,6cyclictest3501986-21kworker/2:023:33:362
3508528995040,6cyclictest3501986-21kworker/2:019:10:552
3508528995040,6cyclictest3501986-21kworker/2:019:10:542
3508528995031,12cyclictest3467602-21systemd-journal21:50:002
350852499507,8cyclictest23-21ksoftirqd/100:16:561
350852499507,10cyclictest23-21ksoftirqd/120:14:591
350852499506,8cyclictest23-21ksoftirqd/121:56:091
350852499506,8cyclictest23-21ksoftirqd/121:56:081
350852499506,7cyclictest23-21ksoftirqd/119:55:011
350852499506,7cyclictest23-21ksoftirqd/119:55:001
350852499505,8cyclictest23-21ksoftirqd/120:05:021
350852499505,7cyclictest23-21ksoftirqd/122:05:551
350852499504,13cyclictest23-21ksoftirqd/122:22:181
350853299497,7cyclictest37-21ksoftirqd/300:15:503
350853299497,10cyclictest3545258-21ssh22:20:393
350853299496,9cyclictest37-21ksoftirqd/323:52:153
350853299496,9cyclictest37-21ksoftirqd/322:11:283
350853299496,8cyclictest37-21ksoftirqd/321:10:433
3508528994941,5cyclictest3542623-21kworker/2:223:07:012
3508528994941,5cyclictest3542623-21kworker/2:222:45:502
3508528994941,5cyclictest3501986-21kworker/2:022:40:422
3508528994941,5cyclictest3501986-21kworker/2:019:59:122
3508528994940,5cyclictest3501986-21kworker/2:023:40:232
3508528994940,5cyclictest3501986-21kworker/2:022:22:202
3508528994940,5cyclictest3501986-21kworker/2:022:13:382
3508528994939,7cyclictest3542623-21kworker/2:200:06:392
3508528994935,10cyclictest1-21systemd20:25:002
350852499494,9cyclictest23-21ksoftirqd/123:08:181
350852499493,9cyclictest23-21ksoftirqd/100:10:531
3508524994918,9cyclictest23-21ksoftirqd/122:50:261
3508524994911,9cyclictest0-21swapper/121:30:011
350853299486,8cyclictest37-21ksoftirqd/300:10:213
350853299486,7cyclictest37-21ksoftirqd/300:31:333
350853299485,10cyclictest37-21ksoftirqd/323:42:543
3508532994828,4cyclictest0-21swapper/319:30:033
3508528994840,5cyclictest3598645-21kworker/2:000:34:072
3508528994840,5cyclictest3542623-21kworker/2:223:52:272
3508528994840,5cyclictest3501986-21kworker/2:020:10:022
3508528994840,5cyclictest3501986-21kworker/2:020:10:012
3508528994839,6cyclictest3542623-21kworker/2:223:19:432
350852499488,6cyclictest23-21ksoftirqd/122:14:261
350852499487,10cyclictest23-21ksoftirqd/121:14:271
350852499486,6cyclictest23-21ksoftirqd/100:09:551
350852499486,12cyclictest3520325-21ssh21:22:571
350852499485,8cyclictest23-21ksoftirqd/122:01:251
3508524994829,14cyclictest1-21systemd21:55:011
350852499481,3cyclictest0-21swapper/119:30:031
3508524994812,6cyclictest131rcu_preempt00:21:211
350853299478,5cyclictest131rcu_preempt20:10:023
350853299478,5cyclictest131rcu_preempt20:10:023
350853299475,8cyclictest37-21ksoftirqd/319:55:023
350853299475,8cyclictest37-21ksoftirqd/319:55:013
3508532994737,7cyclictest361rcuc/321:50:283
3508532994711,9cyclictest37-21ksoftirqd/319:10:013
350852899474,3cyclictest131rcu_preempt20:45:012
350852899473,3cyclictest131rcu_preempt22:18:412
350852899473,3cyclictest131rcu_preempt22:18:402
350852499476,6cyclictest23-21ksoftirqd/120:25:011
350852499473,23cyclictest3508954-21kworker/u8:019:20:021
350852499472,10cyclictest23-21ksoftirqd/121:00:021
3508528994638,5cyclictest3501986-21kworker/2:021:00:032
3508528994638,4cyclictest3501986-21kworker/2:022:25:472
3508528994615,9cyclictest30-21ksoftirqd/219:55:022
3508528994615,9cyclictest30-21ksoftirqd/219:55:022
3508532994510,4cyclictest131rcu_preempt19:44:583
3508528994538,5cyclictest3501986-21kworker/2:020:55:002
3508528994537,6cyclictest3501986-21kworker/2:019:50:012
350852499451,11cyclictest23-21ksoftirqd/120:30:021
350853299441,10cyclictest37-21ksoftirqd/320:30:003
3508528994436,5cyclictest3501986-21kworker/2:019:20:022
350852499449,10cyclictest0-21swapper/120:35:261
350853299434,9cyclictest37-21ksoftirqd/319:23:583
3508532994334,7cyclictest37-21ksoftirqd/319:50:023
3508532994331,9cyclictest3511797-21sshd20:11:563
3508532994329,10cyclictest3467602-21systemd-journal20:35:013
3508532994320,10cyclictest37-21ksoftirqd/320:25:013
3508528994335,5cyclictest3501986-21kworker/2:020:03:592
350852499436,7cyclictest23-21ksoftirqd/120:48:571
350852499436,7cyclictest23-21ksoftirqd/120:48:561
350852499433,11cyclictest23-21ksoftirqd/119:47:041
3508524994310,8cyclictest0-21swapper/119:41:221
350852499429,8cyclictest0-21swapper/119:59:461
350853299417,5cyclictest131rcu_preempt19:20:003
350853299414,8cyclictest37-21ksoftirqd/319:15:013
350853299414,8cyclictest37-21ksoftirqd/319:15:013
3508532994127,10cyclictest275-21dbus-daemon20:00:013
3508532994120,16cyclictest361rcuc/321:10:013
3508528994128,9cyclictest3467602-21systemd-journal20:30:582
3508528994119,15cyclictest275-21dbus-daemon19:25:012
350852499416,10cyclictest23-21ksoftirqd/120:45:011
350852499414,6cyclictest23-21ksoftirqd/121:06:191
350852499413,9cyclictest23-21ksoftirqd/119:35:011
3508524994125,11cyclictest3467602-21systemd-journal20:20:001
3508524994125,11cyclictest3467602-21systemd-journal20:19:591
3508532994033,5cyclictest37-21ksoftirqd/321:00:003
3508528994033,4cyclictest3501986-21kworker/2:020:10:052
350852499406,6cyclictest23-21ksoftirqd/119:18:561
3508524994024,12cyclictest3467602-21systemd-journal20:53:581
350853299385,9cyclictest37-21ksoftirqd/321:03:573
3508532993827,8cyclictest3509726-21sshd19:31:563
350853299382,10cyclictest37-21ksoftirqd/320:05:003
350852899388,6cyclictest131rcu_preempt20:26:502
350852499383,8cyclictest23-21ksoftirqd/119:12:001
350852499383,8cyclictest23-21ksoftirqd/119:11:591
3508524993823,11cyclictest275-21dbus-daemon20:10:011
3508524993823,11cyclictest275-21dbus-daemon20:10:001
3508524993823,11cyclictest23-21ksoftirqd/121:05:001
3508524993810,7cyclictest0-21swapper/119:36:071
3508524993810,7cyclictest0-21swapper/119:36:071
3508528993729,5cyclictest3501986-21kworker/2:019:32:242
3508528993726,8cyclictest3514416-21sshd21:02:582
350852899365,8cyclictest30-21ksoftirqd/219:28:032
3508528993628,5cyclictest3501986-21kworker/2:020:36:122
350853299350,7cyclictest150-21pr/ttyS220:18:393
350853299350,7cyclictest150-21pr/ttyS220:18:393
350852899352,9cyclictest30-21ksoftirqd/219:39:352
350852899352,9cyclictest30-21ksoftirqd/219:39:342
3508528993426,5cyclictest3501986-21kworker/2:020:20:032
3508528993426,5cyclictest3501986-21kworker/2:020:20:022
3508532993322,8cyclictest37-21ksoftirqd/320:55:013
3508524993320,9cyclictest23-21ksoftirqd/119:10:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional