You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-30 - 14:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot3s.osadl.org (updated Tue Jun 30, 2026 00:45:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
290130699782,47cyclictest16950irq/224-800000022:50:030
290132099723,11cyclictest131rcu_preempt20:35:013
2901311996424,10cyclictest131rcu_preempt22:30:481
2901311996424,10cyclictest131rcu_preempt22:30:471
2901311996423,11cyclictest2931486-21cat22:00:031
2901311996417,7cyclictest23-21ksoftirqd/123:59:061
2901311996414,9cyclictest23-21ksoftirqd/100:35:011
2901311996414,9cyclictest23-21ksoftirqd/100:35:001
2901316996322,7cyclictest30-21ksoftirqd/222:29:092
2901311996332,7cyclictest23-21ksoftirqd/121:38:051
2901311996332,7cyclictest23-21ksoftirqd/121:38:041
290130699635,36cyclictest131rcu_preempt21:59:390
2901320996226,10cyclictest37-21ksoftirqd/321:59:573
2901316996216,11cyclictest30-21ksoftirqd/219:40:032
2901316996216,11cyclictest30-21ksoftirqd/219:40:022
290131199625,9cyclictest23-21ksoftirqd/122:15:541
2901311996125,10cyclictest23-21ksoftirqd/100:27:521
2901311996113,4cyclictest131rcu_preempt21:18:301
2901311996111,7cyclictest23-21ksoftirqd/121:45:001
2901306996117,39cyclictest275-21dbus-daemon23:45:010
2901316996040,9cyclictest30-21ksoftirqd/223:17:432
2901316996014,9cyclictest30-21ksoftirqd/200:15:012
2901311996020,8cyclictest23-21ksoftirqd/122:59:511
2901311996019,11cyclictest23-21ksoftirqd/123:13:311
2901320995914,6cyclictest131rcu_preempt22:44:033
290131699592,6cyclictest30-21ksoftirqd/200:40:012
2901316995913,10cyclictest30-21ksoftirqd/221:12:172
290131199597,9cyclictest23-21ksoftirqd/123:20:011
2901311995915,8cyclictest23-21ksoftirqd/100:38:061
2901311995913,8cyclictest23-21ksoftirqd/121:31:521
290130699596,28cyclictest12-21ksoftirqd/022:47:220
2901306995915,5cyclictest131rcu_preempt23:30:420
2901316995839,14cyclictest1-21systemd20:55:002
2901316995835,13cyclictest30-21ksoftirqd/223:12:382
2901316995811,5cyclictest131rcu_preempt23:00:002
2901311995816,11cyclictest23-21ksoftirqd/120:05:001
2901316995717,8cyclictest30-21ksoftirqd/223:35:002
290131199578,6cyclictest0-21swapper/122:00:141
2901311995717,9cyclictest131rcu_preempt20:30:011
2901306995725,16cyclictest2950395-21ssh22:40:010
2901306995722,31cyclictest12-21ksoftirqd/022:21:010
2901306995722,31cyclictest12-21ksoftirqd/022:21:000
2901306995713,7cyclictest12-21ksoftirqd/023:56:410
2901320995636,15cyclictest275-21dbus-daemon23:20:003
2901320995612,23cyclictest0-21swapper/323:25:033
2901320995611,5cyclictest131rcu_preempt22:13:533
2901316995615,10cyclictest30-21ksoftirqd/223:48:282
2901316995611,8cyclictest30-21ksoftirqd/223:28:222
290131699561,12cyclictest30-21ksoftirqd/222:48:072
2901316995610,7cyclictest30-21ksoftirqd/223:21:232
290131199567,8cyclictest23-21ksoftirqd/123:29:101
290131199565,14cyclictest1-21systemd00:20:011
2901311995618,8cyclictest23-21ksoftirqd/100:05:361
2901311995614,8cyclictest23-21ksoftirqd/122:15:031
2901311995614,10cyclictest23-21ksoftirqd/122:22:481
2901311995614,10cyclictest23-21ksoftirqd/122:22:471
2901306995628,13cyclictest2978014-21ssh23:37:410
2901306995627,14cyclictest2986122-21ssh23:54:520
2901306995627,13cyclictest2959262-21ssh22:58:250
2901320995510,6cyclictest131rcu_preempt22:38:173
290131699558,4cyclictest131rcu_preempt22:36:002
290131699557,10cyclictest30-21ksoftirqd/223:00:212
290131699555,9cyclictest30-21ksoftirqd/200:30:002
2901316995518,9cyclictest131rcu_preempt21:25:002
2901316995515,9cyclictest30-21ksoftirqd/221:51:302
2901316995514,9cyclictest30-21ksoftirqd/221:59:392
290131699551,15cyclictest30-21ksoftirqd/200:00:022
2901316995510,8cyclictest30-21ksoftirqd/223:53:272
290131199559,11cyclictest23-21ksoftirqd/122:54:391
290131199556,9cyclictest23-21ksoftirqd/121:13:421
2901311995546,6cyclictest23-21ksoftirqd/123:33:311
290131199553,16cyclictest2994956-21ssh00:13:121
290130699553,5cyclictest131rcu_preempt19:45:010
2901306995517,33cyclictest16850irq/134-800000023:14:590
290130699551,50cyclictest2933976-21ssh22:05:070
2901320995411,5cyclictest131rcu_preempt23:40:403
2901320995410,6cyclictest131rcu_preempt23:53:503
290131699546,7cyclictest30-21ksoftirqd/220:15:012
290131699545,8cyclictest30-21ksoftirqd/221:40:012
290131699545,8cyclictest30-21ksoftirqd/221:40:002
2901316995437,13cyclictest1-21systemd22:45:002
290131199548,9cyclictest2982565-21ssh23:47:111
290131199544,9cyclictest23-21ksoftirqd/122:42:371
290131199541,12cyclictest23-21ksoftirqd/123:55:011
2901306995429,10cyclictest2916221-21ssh21:28:090
2901306995427,12cyclictest2951248-21ssh22:41:310
2901306995426,12cyclictest3006682-21ssh00:37:480
2901306995425,13cyclictest2936485-21ssh22:10:230
290130699542,36cyclictest2983051-21ssh23:48:110
2901306995418,32cyclictest0-21swapper/000:30:580
2901306995418,32cyclictest0-21swapper/000:30:570
2901306995411,36cyclictest16850irq/134-800000021:15:010
290132099539,6cyclictest131rcu_preempt00:18:363
2901320995314,9cyclictest131rcu_preempt19:45:023
2901320995310,6cyclictest131rcu_preempt23:37:323
290131699535,7cyclictest30-21ksoftirqd/223:41:252
290131699535,10cyclictest30-21ksoftirqd/220:50:032
2901316995331,9cyclictest30-21ksoftirqd/222:08:152
290131699533,15cyclictest30-21ksoftirqd/221:27:042
2901316995317,8cyclictest30-21ksoftirqd/222:00:432
2901316995313,9cyclictest30-21ksoftirqd/221:48:012
290131199535,12cyclictest23-21ksoftirqd/121:55:011
290131199535,11cyclictest23-21ksoftirqd/121:21:361
290131199535,10cyclictest23-21ksoftirqd/123:23:251
2901311995315,8cyclictest23-21ksoftirqd/121:28:491
2901311995314,8cyclictest0-21swapper/120:10:011
2901311995314,8cyclictest0-21swapper/120:10:001
2901311995310,8cyclictest23-21ksoftirqd/123:42:331
2901311995310,5cyclictest131rcu_preempt00:20:161
2901311995310,11cyclictest23-21ksoftirqd/123:10:021
290130699532,8cyclictest12-21ksoftirqd/000:07:580
2901306995327,11cyclictest2972747-21ssh23:26:220
2901306995326,12cyclictest2918583-21ssh21:33:020
2901306995326,12cyclictest2912720-21ssh21:20:310
290132099529,6cyclictest131rcu_preempt00:11:573
290132099529,5cyclictest131rcu_preempt21:29:553
290132099528,8cyclictest37-21ksoftirqd/320:38:403
290132099525,10cyclictest37-21ksoftirqd/323:06:403
2901320995214,6cyclictest131rcu_preempt00:25:013
2901320995213,6cyclictest131rcu_preempt19:31:413
2901320995212,5cyclictest131rcu_preempt21:37:373
2901320995212,5cyclictest131rcu_preempt21:37:373
2901320995212,4cyclictest131rcu_preempt23:04:343
2901320995210,6cyclictest131rcu_preempt21:50:563
290131699527,11cyclictest30-21ksoftirqd/222:16:162
290131699525,9cyclictest30-21ksoftirqd/200:15:502
2901316995215,8cyclictest30-21ksoftirqd/222:34:112
2901316995215,8cyclictest30-21ksoftirqd/222:34:112
2901316995215,5cyclictest131rcu_preempt00:32:562
2901316995215,5cyclictest131rcu_preempt00:32:562
290131199527,9cyclictest23-21ksoftirqd/122:07:021
2901311995219,10cyclictest23-21ksoftirqd/119:10:021
2901311995210,10cyclictest23-21ksoftirqd/123:03:401
290130699527,29cyclictest131rcu_preempt00:22:010
2901306995238,11cyclictest16950irq/224-800000021:40:010
2901306995238,11cyclictest16950irq/224-800000021:40:000
2901306995228,10cyclictest2989025-21ssh00:00:260
2901306995228,10cyclictest2946479-21ssh22:31:300
2901306995228,10cyclictest2946479-21ssh22:31:290
2901306995224,13cyclictest2970508-21ssh23:21:420
2901306995224,12cyclictest2969289-21ssh23:19:330
290130699521,10cyclictest12-21ksoftirqd/000:17:240
290132099518,6cyclictest131rcu_preempt21:24:573
290132099515,11cyclictest37-21ksoftirqd/323:57:253
2901320995123,19cyclictest37-21ksoftirqd/322:35:023
2901320995123,19cyclictest37-21ksoftirqd/322:35:013
2901320995111,5cyclictest131rcu_preempt23:26:273
2901320995111,5cyclictest131rcu_preempt00:06:593
2901320995111,17cyclictest131rcu_preempt00:36:363
2901320995110,6cyclictest131rcu_preempt23:12:303
2901320995110,6cyclictest131rcu_preempt22:57:263
2901320995110,5cyclictest131rcu_preempt23:31:343
2901320995110,5cyclictest131rcu_preempt21:31:013
290131699515,7cyclictest30-21ksoftirqd/221:34:442
290131699515,10cyclictest30-21ksoftirqd/222:52:142
290131699514,9cyclictest30-21ksoftirqd/219:50:022
2901316995113,11cyclictest30-21ksoftirqd/223:37:422
290131699511,13cyclictest30-21ksoftirqd/200:23:052
290131199519,7cyclictest23-21ksoftirqd/122:49:521
290131199518,10cyclictest23-21ksoftirqd/119:35:011
290131199514,9cyclictest23-21ksoftirqd/123:36:541
290131199514,8cyclictest23-21ksoftirqd/120:00:021
290131199514,14cyclictest2945804-21cat22:30:011
290131199513,7cyclictest23-21ksoftirqd/121:45:591
290131199512,10cyclictest23-21ksoftirqd/119:15:021
290131199512,10cyclictest23-21ksoftirqd/119:15:011
2901311995114,8cyclictest0-21swapper/121:10:011
2901311995110,7cyclictest23-21ksoftirqd/122:36:171
2901306995114,32cyclictest16850irq/134-800000021:45:470
2901320995011,5cyclictest131rcu_preempt21:16:533
290131699506,7cyclictest30-21ksoftirqd/222:11:572
290131699504,8cyclictest30-21ksoftirqd/200:03:502
290131699501,8cyclictest2907231-21munin-run21:05:012
2901316995013,6cyclictest131rcu_preempt21:19:562
290131199505,9cyclictest23-21ksoftirqd/119:45:031
2901311995012,10cyclictest0-21swapper/119:40:011
2901311995012,10cyclictest0-21swapper/119:40:011
2901306995025,9cyclictest2924274-21ssh21:45:010
2901306995024,8cyclictest12-21ksoftirqd/000:29:190
290132099498,6cyclictest131rcu_preempt22:06:343
290132099498,5cyclictest131rcu_preempt22:54:273
2901320994911,6cyclictest131rcu_preempt21:42:183
2901320994910,6cyclictest131rcu_preempt22:15:423
2901320994910,6cyclictest131rcu_preempt21:47:223
290131699498,8cyclictest30-21ksoftirqd/219:30:012
290131699498,8cyclictest30-21ksoftirqd/219:30:002
290131699496,6cyclictest2905155-21latency_hist20:25:032
290131699496,6cyclictest2905155-21latency_hist20:25:022
290131699495,9cyclictest30-21ksoftirqd/221:40:222
2901316994912,11cyclictest30-21ksoftirqd/200:05:252
290131199497,8cyclictest23-21ksoftirqd/120:15:001
290131199493,8cyclictest23-21ksoftirqd/100:02:191
290131199492,10cyclictest23-21ksoftirqd/121:05:021
290131199491,13cyclictest2905212-21date20:25:011
290131199491,13cyclictest2905212-21date20:25:011
2901306994932,13cyclictest1-21systemd20:10:010
2901306994932,13cyclictest1-21systemd20:10:000
290132099488,5cyclictest131rcu_preempt00:00:193
2901320994819,18cyclictest37-21ksoftirqd/322:49:403
290132099481,7cyclictest0-21swapper/322:01:173
2901320994810,7cyclictest131rcu_preempt00:29:253
2901320994810,5cyclictest131rcu_preempt22:23:453
2901320994810,5cyclictest131rcu_preempt22:23:453
290131699486,7cyclictest30-21ksoftirqd/222:21:162
290131699486,7cyclictest30-21ksoftirqd/222:21:162
2901316994840,5cyclictest2894437-21kworker/2:019:56:142
2901316994840,5cyclictest2875928-21kworker/2:320:07:562
2901316994840,5cyclictest2875928-21kworker/2:320:07:562
290131699482,4cyclictest131rcu_preempt21:00:002
2901316994823,8cyclictest30-21ksoftirqd/223:05:342
290131199489,6cyclictest23-21ksoftirqd/119:55:021
290131199487,10cyclictest2901926-21date19:20:011
290131199486,8cyclictest23-21ksoftirqd/119:25:011
290131199484,12cyclictest23-21ksoftirqd/120:50:001
290131199483,8cyclictest23-21ksoftirqd/120:20:001
2901311994813,7cyclictest0-21swapper/121:00:011
290130699485,6cyclictest131rcu_preempt23:07:520
290130699484,9cyclictest12-21ksoftirqd/021:16:330
2901306994833,11cyclictest16950irq/224-800000023:05:010
2901306994831,13cyclictest3467602-21systemd-journal21:00:000
2901306994828,15cyclictest275-21dbus-daemon19:20:000
2901306994821,13cyclictest2993952-21ssh00:10:570
2901306994815,5cyclictest131rcu_preempt22:17:170
290132099472,7cyclictest37-21ksoftirqd/322:29:513
2901320994711,5cyclictest131rcu_preempt23:45:413
290131699476,6cyclictest30-21ksoftirqd/220:35:012
290131699474,12cyclictest30-21ksoftirqd/219:35:002
290131699472,11cyclictest30-21ksoftirqd/220:35:402
2901316994712,8cyclictest30-21ksoftirqd/221:10:022
2901311994732,11cyclictest3467602-21systemd-journal19:27:191
2901311994732,11cyclictest3467602-21systemd-journal19:27:181
2901306994739,4cyclictest16950irq/224-800000020:25:020
2901306994739,4cyclictest16950irq/224-800000020:25:010
2901306994728,15cyclictest2931846-21ssh22:00:360
2901306994721,9cyclictest12-21ksoftirqd/020:00:000
2901320994616,14cyclictest37-21ksoftirqd/300:30:323
2901320994616,14cyclictest37-21ksoftirqd/300:30:313
290131699464,7cyclictest2901399-21sed19:10:032
2901316994639,5cyclictest2894437-21kworker/2:020:42:002
2901316994638,5cyclictest2894437-21kworker/2:019:24:172
2901316994631,11cyclictest2904079-21sshd20:00:412
290131199462,8cyclictest23-21ksoftirqd/120:35:001
290131199462,6cyclictest23-21ksoftirqd/119:50:001
290130699463,31cyclictest12-21ksoftirqd/020:35:010
2901306994628,15cyclictest2907600-21latency_hist21:10:010
2901306994618,20cyclictest0-21swapper/021:52:150
2901320994519,15cyclictest37-21ksoftirqd/319:15:013
2901320994519,15cyclictest37-21ksoftirqd/319:15:013
290132099451,8cyclictest0-21swapper/320:25:003
290132099451,8cyclictest0-21swapper/320:25:003
290132099451,6cyclictest131rcu_preempt20:55:023
2901320994515,19cyclictest37-21ksoftirqd/321:10:383
2901320994513,6cyclictest131rcu_preempt19:40:003
2901320994513,6cyclictest131rcu_preempt19:40:003
2901320994512,21cyclictest131rcu_preempt20:50:023
2901320994510,6cyclictest131rcu_preempt21:00:023
2901320994510,6cyclictest131rcu_preempt19:55:023
290131699457,8cyclictest30-21ksoftirqd/219:20:012
290131199453,10cyclictest23-21ksoftirqd/120:45:031
290131199451,10cyclictest23-21ksoftirqd/120:50:021
2901306994528,14cyclictest2902520-21sshd19:32:190
290132099449,6cyclictest131rcu_preempt20:15:013
290132099441,11cyclictest2907513-21systemd-run21:10:023
290131699444,8cyclictest30-21ksoftirqd/220:30:002
2901316994411,9cyclictest30-21ksoftirqd/219:45:012
2901306994438,4cyclictest16950irq/224-800000020:45:000
2901306994433,7cyclictest12-21ksoftirqd/019:40:010
2901306994433,7cyclictest12-21ksoftirqd/019:40:000
2901306994429,12cyclictest2944982-21ssh22:28:310
2901306994428,8cyclictest12-21ksoftirqd/019:12:350
2901306994428,8cyclictest12-21ksoftirqd/019:12:340
290130699442,7cyclictest1-21systemd19:55:010
2901320994330,10cyclictest37-21ksoftirqd/320:09:413
2901320994330,10cyclictest37-21ksoftirqd/320:09:403
290131199437,26cyclictest2905908-21modprobe20:40:001
2901306994327,13cyclictest2905878-21sshd20:35:410
2901306994313,19cyclictest151rcuc/020:05:020
2901306994310,17cyclictest12-21ksoftirqd/020:28:170
2901320994218,6cyclictest131rcu_preempt20:20:023
2901320994210,4cyclictest131rcu_preempt20:40:413
290131699428,9cyclictest0-21swapper/219:13:222
290131699428,9cyclictest0-21swapper/219:13:222
2901306994217,9cyclictest12-21ksoftirqd/019:50:010
2901306994214,5cyclictest131rcu_preempt19:25:490
2901306994214,5cyclictest131rcu_preempt19:25:480
290132099417,8cyclictest37-21ksoftirqd/319:48:403
290132099416,6cyclictest37-21ksoftirqd/319:17:163
290132099411,7cyclictest0-21swapper/320:25:413
290131699418,8cyclictest0-21swapper/220:20:002
2901316994110,5cyclictest0-21swapper/219:50:242
2901320994028,7cyclictest37-21ksoftirqd/319:30:013
2901320994028,7cyclictest37-21ksoftirqd/319:30:013
2901320994026,10cyclictest275-21dbus-daemon20:00:023
2901320994011,8cyclictest131rcu_preempt19:10:023
2901320994011,5cyclictest131rcu_preempt19:25:023
290130699401,14cyclictest364-21snmpd20:16:050
290132099392,26cyclictest37-21ksoftirqd/320:05:023
290130699396,9cyclictest131rcu_preempt21:04:400
2901306993923,13cyclictest275-21dbus-daemon20:55:010
2901320993710,15cyclictest37-21ksoftirqd/321:01:333
2901306993720,13cyclictest2902030-21sshd19:24:390
290130699371,5cyclictest2906446-21munin-run20:50:000
2901306993525,7cyclictest0-21swapper/019:10:010
2901306993518,13cyclictest3467602-21systemd-journal20:14:400
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional