You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-23 - 22:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Sat May 23, 2026 12:45:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3524594999789,6cyclictest12-21ksoftirqd/011:30:020
3524594998875,7cyclictest3524597-21cat07:10:010
3524594997773,2cyclictest0-21swapper/011:05:030
3524600997017,5cyclictest131rcu_preempt08:20:001
352459499672,7cyclictest121ksoftirqd/011:15:350
3524604996617,10cyclictest30-21ksoftirqd/207:20:002
3524604996610,16cyclictest3532170-21ssh09:13:032
352460099667,7cyclictest23-21ksoftirqd/110:00:001
3524594996613,48cyclictest12-21ksoftirqd/010:05:010
352459499653,5cyclictest131rcu_preempt11:22:060
3524600996414,5cyclictest131rcu_preempt09:15:011
352459499647,36cyclictest131rcu_preempt11:47:060
352459499647,36cyclictest131rcu_preempt11:47:060
352460499639,16cyclictest3593967-21ssh11:21:432
352460499638,16cyclictest3544823-21ssh09:39:252
352460499636,16cyclictest3546716-21ssh09:43:142
3524604996313,10cyclictest30-21ksoftirqd/209:30:012
3524604996313,10cyclictest30-21ksoftirqd/209:30:002
3524604996312,6cyclictest30-21ksoftirqd/212:07:452
3524604996312,6cyclictest30-21ksoftirqd/212:07:452
3524606996212,5cyclictest131rcu_preempt10:55:073
3524604996220,8cyclictest30-21ksoftirqd/209:00:012
3524604996216,9cyclictest30-21ksoftirqd/210:00:032
3524604996216,10cyclictest30-21ksoftirqd/210:59:462
3524604996214,9cyclictest30-21ksoftirqd/210:14:182
3524594996212,5cyclictest131rcu_preempt08:20:010
352460499618,15cyclictest3559103-21ssh10:09:102
352460499612,21cyclictest3601251-21ssh11:37:072
352460499612,21cyclictest3601251-21ssh11:37:062
3524604996118,8cyclictest30-21ksoftirqd/209:52:242
3524604996117,9cyclictest30-21ksoftirqd/212:35:562
3524604996116,9cyclictest30-21ksoftirqd/210:30:442
3524604996115,8cyclictest30-21ksoftirqd/211:44:052
3524604996114,10cyclictest30-21ksoftirqd/207:35:022
3524604996114,10cyclictest30-21ksoftirqd/207:35:012
352459499615,6cyclictest12-21ksoftirqd/010:40:020
352460699608,6cyclictest131rcu_preempt12:23:473
3524604996014,10cyclictest30-21ksoftirqd/210:01:472
3524604996013,9cyclictest30-21ksoftirqd/209:16:232
3524604996013,9cyclictest30-21ksoftirqd/209:16:222
3524604996012,7cyclictest30-21ksoftirqd/211:52:222
352460099606,8cyclictest23-21ksoftirqd/107:20:011
3524600996037,10cyclictest23-21ksoftirqd/109:22:201
3524600996011,5cyclictest131rcu_preempt09:41:391
3524594996038,18cyclictest3530772-21diskmemload09:22:000
3524604995918,7cyclictest30-21ksoftirqd/212:16:572
3524604995917,8cyclictest30-21ksoftirqd/210:43:232
3524604995915,8cyclictest30-21ksoftirqd/208:55:022
3524604995914,9cyclictest30-21ksoftirqd/209:33:382
3524604995913,8cyclictest30-21ksoftirqd/212:00:402
3524604995912,9cyclictest30-21ksoftirqd/212:31:472
3524604995912,8cyclictest30-21ksoftirqd/210:38:472
3524600995911,5cyclictest131rcu_preempt11:15:021
352460699587,7cyclictest37-21ksoftirqd/311:28:073
3524606995847,7cyclictest37-21ksoftirqd/310:50:023
3524604995811,7cyclictest30-21ksoftirqd/210:19:062
352460099589,5cyclictest131rcu_preempt09:45:111
352460099589,5cyclictest131rcu_preempt09:45:101
352460099589,4cyclictest131rcu_preempt10:27:461
3524600995812,5cyclictest131rcu_preempt09:52:441
3524594995846,8cyclictest12-21ksoftirqd/012:14:070
352459499583,5cyclictest131rcu_preempt10:07:540
3524594995819,34cyclictest16850irq/134-800000009:30:360
352460699579,5cyclictest131rcu_preempt09:59:403
352460699575,9cyclictest37-21ksoftirqd/309:26:103
352460699575,9cyclictest37-21ksoftirqd/309:26:103
3524606995714,6cyclictest131rcu_preempt11:54:133
3524606995714,5cyclictest131rcu_preempt11:31:553
3524606995712,5cyclictest131rcu_preempt09:32:033
352460499577,15cyclictest3526995-21sed07:55:012
3524604995715,8cyclictest30-21ksoftirqd/210:52:592
3524604995715,8cyclictest30-21ksoftirqd/210:52:582
3524604995714,9cyclictest30-21ksoftirqd/212:20:052
3524604995714,10cyclictest30-21ksoftirqd/211:48:482
3524604995714,10cyclictest30-21ksoftirqd/211:48:482
3524604995712,8cyclictest30-21ksoftirqd/211:12:122
3524604995712,10cyclictest30-21ksoftirqd/211:15:532
3524604995710,11cyclictest30-21ksoftirqd/211:55:252
352460099578,9cyclictest23-21ksoftirqd/111:39:041
352460099578,9cyclictest23-21ksoftirqd/111:39:031
352460099577,7cyclictest23-21ksoftirqd/112:16:261
352460099574,8cyclictest23-21ksoftirqd/112:13:121
352460099573,8cyclictest23-21ksoftirqd/109:35:031
3524600995716,6cyclictest131rcu_preempt12:30:181
3524600995714,6cyclictest131rcu_preempt12:06:081
3524600995714,6cyclictest131rcu_preempt12:06:071
3524600995710,6cyclictest131rcu_preempt11:30:061
3524600995710,5cyclictest131rcu_preempt11:03:481
3524600995710,5cyclictest131rcu_preempt11:03:471
352459499578,5cyclictest131rcu_preempt09:54:190
352459499578,5cyclictest131rcu_preempt09:11:060
3524594995724,10cyclictest12-21ksoftirqd/010:12:590
3524594995723,30cyclictest1-21systemd10:00:020
3524594995721,33cyclictest151rcuc/009:38:190
3524594995720,32cyclictest16850irq/134-800000012:38:270
3524594995718,10cyclictest12-21ksoftirqd/008:10:010
352459499571,4cyclictest0-21swapper/009:40:150
3524594995714,6cyclictest131rcu_preempt10:59:250
352460699568,8cyclictest37-21ksoftirqd/309:15:033
352460699566,6cyclictest131rcu_preempt10:16:083
352460699562,8cyclictest37-21ksoftirqd/312:15:003
3524606995614,5cyclictest131rcu_preempt10:31:253
352460499568,14cyclictest3565076-21ssh10:21:212
352460499568,14cyclictest3565076-21ssh10:21:212
352460099567,8cyclictest23-21ksoftirqd/111:47:531
352460099567,8cyclictest23-21ksoftirqd/111:47:521
352460099567,10cyclictest221rcuc/109:26:261
352460099567,10cyclictest221rcuc/109:26:251
352460099566,10cyclictest23-21ksoftirqd/111:26:251
3524600995612,6cyclictest131rcu_preempt10:49:281
3524600995610,5cyclictest131rcu_preempt11:17:391
3524594995624,14cyclictest131rcu_preempt11:44:350
3524594995623,13cyclictest3628909-21ssh12:35:000
3524594995617,6cyclictest131rcu_preempt10:52:050
3524594995617,6cyclictest131rcu_preempt10:52:050
3524594995611,6cyclictest131rcu_preempt10:29:590
352460699559,5cyclictest131rcu_preempt11:45:153
352460699559,5cyclictest131rcu_preempt11:45:153
352460699555,10cyclictest37-21ksoftirqd/311:36:223
352460699555,10cyclictest37-21ksoftirqd/311:36:223
3524606995544,8cyclictest361rcuc/311:06:013
3524606995511,5cyclictest131rcu_preempt09:44:223
3524606995511,5cyclictest131rcu_preempt09:21:393
3524604995515,9cyclictest30-21ksoftirqd/211:09:092
3524604995515,7cyclictest30-21ksoftirqd/208:25:032
3524604995515,7cyclictest30-21ksoftirqd/208:25:022
3524604995514,8cyclictest30-21ksoftirqd/209:49:272
3524604995514,8cyclictest30-21ksoftirqd/209:49:262
3524604995514,10cyclictest30-21ksoftirqd/212:10:422
3524604995513,9cyclictest30-21ksoftirqd/209:23:432
352460099558,11cyclictest23-21ksoftirqd/109:17:401
352460099558,11cyclictest23-21ksoftirqd/109:17:391
352460099556,10cyclictest23-21ksoftirqd/110:10:311
352460099552,7cyclictest23-21ksoftirqd/111:56:591
3524600995513,6cyclictest131rcu_preempt12:21:011
3524600995511,5cyclictest131rcu_preempt08:00:011
352459499554,9cyclictest12-21ksoftirqd/007:53:060
3524594995520,31cyclictest16850irq/134-800000011:59:550
3524594995520,31cyclictest16850irq/134-800000009:25:160
3524594995520,31cyclictest16850irq/134-800000009:25:160
3524594995519,32cyclictest16850irq/134-800000010:15:180
3524594995518,32cyclictest16850irq/134-800000012:25:120
3524594995518,32cyclictest16850irq/134-800000012:25:120
3524594995518,32cyclictest16850irq/134-800000011:00:450
3524594995518,32cyclictest16850irq/134-800000011:00:450
352459499551,38cyclictest3608963-21ssh11:53:210
352460699549,6cyclictest131rcu_preempt12:02:153
352460699549,5cyclictest131rcu_preempt10:37:033
352460699547,8cyclictest37-21ksoftirqd/311:16:073
352460699546,5cyclictest131rcu_preempt12:38:183
3524606995416,5cyclictest131rcu_preempt09:36:593
3524606995415,5cyclictest131rcu_preempt08:40:003
3524606995414,6cyclictest131rcu_preempt12:33:023
3524606995411,5cyclictest131rcu_preempt12:07:063
3524606995411,5cyclictest131rcu_preempt12:07:063
3524606995410,6cyclictest131rcu_preempt11:40:063
3524604995414,7cyclictest30-21ksoftirqd/211:03:132
3524604995414,7cyclictest30-21ksoftirqd/211:03:122
3524600995411,5cyclictest131rcu_preempt12:27:161
3524600995411,5cyclictest131rcu_preempt12:27:161
3524600995410,6cyclictest131rcu_preempt10:07:211
352459499546,6cyclictest131rcu_preempt11:10:390
3524594995428,11cyclictest3534365-21ssh09:17:280
3524594995428,11cyclictest3534365-21ssh09:17:280
3524594995419,31cyclictest16850irq/134-800000010:50:020
3524594995419,30cyclictest16850irq/134-800000012:23:280
3524594995415,6cyclictest131rcu_preempt09:47:070
3524594995415,6cyclictest131rcu_preempt09:47:070
352460699537,8cyclictest37-21ksoftirqd/311:20:063
352460699535,8cyclictest37-21ksoftirqd/310:52:093
352460699535,8cyclictest37-21ksoftirqd/310:52:093
352460699531,9cyclictest37-21ksoftirqd/310:43:243
3524606995311,6cyclictest131rcu_preempt12:27:233
3524606995311,6cyclictest131rcu_preempt12:27:223
3524606995311,6cyclictest131rcu_preempt10:07:003
3524606995310,6cyclictest131rcu_preempt12:15:423
3524606995310,5cyclictest131rcu_preempt10:26:153
3524604995314,8cyclictest30-21ksoftirqd/210:26:432
3524604995312,7cyclictest30-21ksoftirqd/211:28:512
352460099539,4cyclictest131rcu_preempt11:40:051
352460099537,9cyclictest23-21ksoftirqd/110:56:591
352460099537,5cyclictest131rcu_preempt10:40:591
352460099536,5cyclictest131rcu_preempt10:20:171
352460099536,5cyclictest131rcu_preempt10:20:161
352460099534,8cyclictest23-21ksoftirqd/110:01:071
3524600995314,6cyclictest131rcu_preempt11:08:301
3524600995313,5cyclictest131rcu_preempt11:53:191
3524600995313,5cyclictest131rcu_preempt10:15:061
3524600995311,5cyclictest131rcu_preempt10:52:401
3524600995311,5cyclictest131rcu_preempt10:52:401
3524594995317,31cyclictest16850irq/134-800000012:16:070
3524594995312,6cyclictest131rcu_preempt08:50:020
3524594995311,6cyclictest131rcu_preempt12:08:070
3524594995311,6cyclictest131rcu_preempt12:08:060
352460699527,5cyclictest131rcu_preempt09:53:003
3524606995213,6cyclictest131rcu_preempt11:56:303
3524606995213,6cyclictest131rcu_preempt10:02:413
3524606995213,6cyclictest131rcu_preempt09:20:013
3524606995213,6cyclictest131rcu_preempt09:20:003
352460499527,14cyclictest3527405-21munin-run08:05:002
352460499524,14cyclictest3577846-21ssh10:48:172
3524604995213,6cyclictest30-21ksoftirqd/212:25:372
3524604995213,6cyclictest30-21ksoftirqd/212:25:372
352460099524,7cyclictest23-21ksoftirqd/111:24:341
352460099522,10cyclictest23-21ksoftirqd/112:02:351
352460099521,8cyclictest23-21ksoftirqd/107:25:021
3524600995213,6cyclictest131rcu_preempt10:30:031
3524600995213,6cyclictest131rcu_preempt09:34:211
3524594995219,30cyclictest16850irq/134-800000012:03:200
3524594995217,31cyclictest16850irq/134-800000010:20:420
3524594995217,31cyclictest16850irq/134-800000010:20:410
3524594995216,31cyclictest16850irq/134-800000010:33:350
352460699517,5cyclictest131rcu_preempt11:10:513
3524606995114,5cyclictest131rcu_preempt07:40:003
3524606995113,5cyclictest131rcu_preempt08:35:023
3524606995113,5cyclictest131rcu_preempt08:35:013
3524606995110,5cyclictest131rcu_preempt11:01:023
3524606995110,5cyclictest131rcu_preempt11:01:023
3524604995142,5cyclictest3526097-21munin-run07:39:592
3524604995112,7cyclictest30-21ksoftirqd/211:30:482
352460099518,5cyclictest131rcu_preempt10:37:491
352460099517,8cyclictest23-21ksoftirqd/112:36:441
3524600995116,5cyclictest131rcu_preempt08:30:001
3524594995118,30cyclictest16850irq/134-800000011:31:220
3524594995115,6cyclictest131rcu_preempt09:10:020
3524594995115,32cyclictest16850irq/134-800000011:35:530
3524594995115,32cyclictest16850irq/134-800000011:35:530
3524594995115,32cyclictest16850irq/134-800000010:40:070
352460699509,6cyclictest131rcu_preempt10:12:243
3524606995015,5cyclictest131rcu_preempt10:20:493
3524606995015,5cyclictest131rcu_preempt10:20:483
3524606995012,6cyclictest131rcu_preempt09:45:253
3524606995012,6cyclictest131rcu_preempt09:45:243
352460099506,6cyclictest23-21ksoftirqd/108:50:021
352460099506,6cyclictest23-21ksoftirqd/108:50:011
3524594995024,5cyclictest131rcu_preempt08:14:590
352460699497,5cyclictest131rcu_preempt07:53:063
3524606994913,5cyclictest131rcu_preempt08:05:003
3524604994929,14cyclictest309-21in:imuxsock08:20:012
352460099497,10cyclictest23-21ksoftirqd/107:25:011
352459499492,15cyclictest3526097-21munin-run07:40:000
352459499491,13cyclictest12-21ksoftirqd/007:30:030
352460699486,8cyclictest37-21ksoftirqd/308:41:053
3524606994821,6cyclictest361rcuc/308:30:013
3524606994811,5cyclictest131rcu_preempt07:30:013
3524604994832,12cyclictest275-21dbus-daemon07:50:012
352460099487,7cyclictest23-21ksoftirqd/108:10:021
352460099486,10cyclictest23-21ksoftirqd/107:35:021
352460099486,10cyclictest23-21ksoftirqd/107:35:021
3524600994810,5cyclictest131rcu_preempt07:50:011
352460699475,7cyclictest37-21ksoftirqd/309:00:013
352460699471,8cyclictest0-21swapper/308:24:003
352460699471,8cyclictest0-21swapper/308:23:593
3524606994714,6cyclictest131rcu_preempt08:15:013
3524606994711,6cyclictest131rcu_preempt09:09:073
3524604994710,7cyclictest30-21ksoftirqd/208:00:012
352460099477,7cyclictest23-21ksoftirqd/108:05:011
352460099476,10cyclictest23-21ksoftirqd/108:25:021
352460099476,10cyclictest23-21ksoftirqd/108:25:011
352459499477,5cyclictest131rcu_preempt08:55:010
352459499476,5cyclictest131rcu_preempt08:40:000
352459499473,4cyclictest131rcu_preempt08:05:000
3524594994712,6cyclictest131rcu_preempt08:50:020
3524606994633,9cyclictest361rcuc/308:50:023
3524606994633,9cyclictest361rcuc/308:50:023
352460699461,7cyclictest0-21swapper/307:20:593
3524604994638,5cyclictest3525012-21kworker/2:007:25:032
3524604994631,11cyclictest3530758-21cron09:09:592
352460099468,7cyclictest23-21ksoftirqd/107:10:021
352460099465,7cyclictest23-21ksoftirqd/109:05:021
352460099465,7cyclictest23-21ksoftirqd/109:05:011
352460099465,7cyclictest23-21ksoftirqd/107:40:001
352460099462,10cyclictest23-21ksoftirqd/107:50:021
3524600994612,5cyclictest131rcu_preempt08:35:071
3524600994610,6cyclictest131rcu_preempt08:45:021
3524594994611,5cyclictest131rcu_preempt08:24:070
3524594994611,5cyclictest131rcu_preempt08:24:060
352460699451,8cyclictest0-21swapper/308:52:413
3524606994513,5cyclictest131rcu_preempt07:58:443
352460699445,6cyclictest37-21ksoftirqd/307:48:063
352460699444,9cyclictest37-21ksoftirqd/309:05:033
352460699444,9cyclictest37-21ksoftirqd/309:05:023
352460699441,8cyclictest0-21swapper/307:10:573
352460499444,28cyclictest3528713-21unin-run08:30:012
3524604994428,12cyclictest3526351-21sshd07:44:052
3524604994425,14cyclictest309-21in:imuxsock08:05:022
352460099448,5cyclictest131rcu_preempt09:06:061
352460099447,5cyclictest131rcu_preempt08:10:031
352460099441,6cyclictest0-21swapper/108:58:061
352459499449,5cyclictest131rcu_preempt07:15:020
3524594994434,7cyclictest151rcuc/009:00:000
352459499442,5cyclictest131rcu_preempt09:05:030
352459499442,5cyclictest131rcu_preempt09:05:020
352460499436,24cyclictest3524794-21munin-plugin-st07:15:012
3524604994316,22cyclictest3527878-21sshd08:10:452
3524604994311,26cyclictest311-21rs:main2
352460099438,6cyclictest131rcu_preempt07:40:061
352459499436,8cyclictest12-21ksoftirqd/007:57:050
3524594994334,7cyclictest12-21ksoftirqd/007:48:440
3524594994334,6cyclictest151rcuc/008:33:490
3524594994334,6cyclictest151rcuc/008:33:480
3524594994319,20cyclictest151rcuc/008:30:020
3524606994226,12cyclictest37-21ksoftirqd/307:20:013
352460699421,7cyclictest0-21swapper/308:19:223
3524604994225,13cyclictest3467602-21systemd-journal07:28:032
352460099424,9cyclictest23-21ksoftirqd/107:10:131
352460099421,14cyclictest0-21swapper/108:30:321
352460099421,14cyclictest0-21swapper/108:30:311
352459499422,6cyclictest3525061-21modprobe07:20:000
352459499421,7cyclictest275-21dbus-daemon07:45:000
352460699411,6cyclictest0-21swapper/307:32:153
352460699411,6cyclictest0-21swapper/307:32:143
352460699411,10cyclictest37-21ksoftirqd/308:10:033
3524604994119,16cyclictest3529544-21wc08:45:022
352460099413,9cyclictest23-21ksoftirqd/108:55:011
352459499418,5cyclictest131rcu_preempt07:20:020
3524594994128,10cyclictest3525819-21sshd07:34:070
3524594994128,10cyclictest3525819-21sshd07:34:060
3524604994021,15cyclictest364-21snmpd09:00:262
3524604994021,15cyclictest364-21snmpd09:00:252
3524604993927,9cyclictest3529035-21cut08:35:022
3524604993927,9cyclictest3529035-21cut08:35:012
3524604993919,16cyclictest364-21snmpd08:45:222
3524604993919,16cyclictest364-21snmpd08:45:212
352460499381,11cyclictest3524671-21cat07:10:022
3524594993830,6cyclictest12-21ksoftirqd/008:45:000
352460699368,4cyclictest131rcu_preempt07:42:053
352460699351,30cyclictest3524717-21cat07:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional