You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-05 - 23:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Tue May 05, 2026 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11748996644,14cyclictest35966irq/58-eth011:30:133
1165699606,33cyclictest10741-21ssh09:35:140
1168999594,41cyclictest3319-21ssh10:35:121
1165699572,11cyclictest630-21Async09:36:310
1173299551,35cyclictest25084-21ssh09:25:142
11689995215,26cyclictest21-21ksoftirqd/112:30:131
11732994929,11cyclictest131rcu_preempt09:10:132
1168999492,40cyclictest24978-21ssh11:05:131
1165699465,37cyclictest1571-21ssh11:10:110
1168999442,28cyclictest24829-21ssh12:35:121
11732994322,14cyclictest0-21swapper/210:35:122
11689994320,10cyclictest131rcu_preempt09:30:121
1165699412,36cyclictest12393-21ssh10:40:140
1168999402,28cyclictest21-21ksoftirqd/107:45:131
1165699404,26cyclictest26725-21ssh10:30:130
1168999392,30cyclictest5280-21ssh09:55:131
1168999381,31cyclictest12384-21ssh12:10:131
1168999381,31cyclictest12384-21ssh12:10:131
1168999374,23cyclictest30391-21ssh10:50:121
1174899363,17cyclictest35966irq/58-eth010:50:133
1165699360,28cyclictest0-21swapper/007:35:120
1174899351,20cyclictest35966irq/58-eth012:20:133
1168999354,24cyclictest28534-21ssh11:25:131
1165699352,27cyclictest32119-21ssh10:15:100
1174899343,14cyclictest35966irq/58-eth007:45:133
11689993421,8cyclictest23201-21ssh11:40:121
1174899332,19cyclictest0-21swapper/309:55:133
1165699332,25cyclictest16018-21ssh09:20:140
1174899322,14cyclictest35966irq/58-eth010:15:093
1168999325,7cyclictest131rcu_preempt07:15:131
1165699325,24cyclictest3363-21ssh12:05:120
1165699325,24cyclictest3363-21ssh12:05:120
1174899312,16cyclictest35966irq/58-eth008:05:143
1174899312,16cyclictest35966irq/58-eth008:05:133
11748993120,4cyclictest0-21swapper/311:10:143
11748993115,8cyclictest35966irq/58-eth010:40:143
1168999311,22cyclictest21365-21ssh10:45:121
1165699311,23cyclictest32213-21rm11:45:130
1174899303,22cyclictest35966irq/58-eth011:15:113
1173299301,4cyclictest0-21swapper/211:05:132
1165699302,22cyclictest672-21ssh12:39:360
11656993018,6cyclictest0-21swapper/012:15:110
1174899292,16cyclictest35966irq/58-eth010:20:113
1173299295,15cyclictest27346irq/40-dwc2_hso09:45:132
1173299294,11cyclictest0-21swapper/208:25:132
1173299291,11cyclictest5149-21ssh11:30:122
11689992916,7cyclictest0-21swapper/109:25:151
1173299284,17cyclictest17795-21ssh11:55:142
1173299282,20cyclictest0-21swapper/208:05:152
1173299282,20cyclictest0-21swapper/208:05:152
1173299282,14cyclictest19578-21ssh11:20:122
1173299282,14cyclictest19578-21ssh11:20:122
11689992819,7cyclictest21-21ksoftirqd/112:15:121
1174899273,15cyclictest0-21swapper/311:25:143
1173299279,10cyclictest27787-21kworker/2:111:25:132
1173299272,19cyclictest0-21swapper/208:40:152
1173299272,11cyclictest0-21swapper/207:25:162
1173299271,23cyclictest30393-21ssh12:20:122
1168999272,19cyclictest8757-21ssh10:20:121
1165699272,21cyclictest26785-21ssh12:00:130
1165699272,17cyclictest17810-21ssh11:35:240
11656992710,13cyclictest19587-21diskmemload11:20:130
11656992710,13cyclictest19587-21diskmemload11:20:120
1174899265,15cyclictest35966irq/58-eth010:30:133
1174899264,18cyclictest35966irq/58-eth011:35:123
1173299263,16cyclictest27346irq/40-dwc2_hso08:45:132
1173299262,9cyclictest0-21swapper/209:30:132
1173299262,20cyclictest0-21swapper/211:45:132
1173299262,12cyclictest5289-21ssh09:55:142
1173299262,10cyclictest0-21swapper/208:55:142
1173299261,22cyclictest27677-21grep08:00:152
1168999265,15cyclictest21-21ksoftirqd/111:30:131
1168999261,18cyclictest18700-21ssh10:05:131
1173299252,19cyclictest0-21swapper/208:35:122
11732992513,9cyclictest21389-21ssh12:31:172
11732992512,9cyclictest17710-21/usr/sbin/munin10:25:242
11732992512,9cyclictest17710-21/usr/sbin/munin10:25:242
1173299251,21cyclictest29956-21ssh09:10:012
1173299240,17cyclictest12303-21ls11:50:242
1168999243,15cyclictest201rcuc/110:25:131
1168999243,15cyclictest201rcuc/110:25:121
1168999242,18cyclictest3396-21ssh11:45:251
11689992413,5cyclictest211ksoftirqd/110:40:141
1168999241,16cyclictest26715-21ssh10:30:121
11656992411,3cyclictest131rcu_preempt09:45:290
1174899233,13cyclictest0-21swapper/312:30:143
11748992318,3cyclictest321rcuc/307:20:023
11748992317,4cyclictest321rcuc/310:57:583
11748992311,7cyclictest0-21swapper/312:00:123
1173299234,9cyclictest30396-21ssh10:50:132
1173299232,6cyclictest0-21swapper/211:40:142
1173299232,16cyclictest0-21swapper/212:36:352
1173299232,11cyclictest10658-21sh10:55:262
1173299231,6cyclictest0-21swapper/212:15:112
11732992312,9cyclictest16361-21ssh11:16:402
1173299231,13cyclictest30707-21ssh12:00:282
1168999232,16cyclictest10920-21apt-get08:50:121
1168999231,17cyclictest15762-21apt-get08:55:131
1165699232,14cyclictest6978-21ssh12:25:140
1174899222,6cyclictest0-21swapper/310:45:123
11748992215,5cyclictest0-21swapper/311:23:083
11748992215,5cyclictest0-21swapper/311:23:073
11748992213,7cyclictest0-21swapper/312:39:473
1173299228,5cyclictest17324-21kthreadcore11:00:202
1173299224,14cyclictest27346irq/40-dwc2_hso07:50:122
1173299223,10cyclictest11032-21fschecks_count08:50:162
1173299222,14cyclictest27346irq/40-dwc2_hso07:30:122
11732992211,9cyclictest9042-21ssh11:13:392
1173299220,12cyclictest2364266ssh09:44:512
1168999227,4cyclictest131rcu_preempt11:50:131
1168999224,13cyclictest21-21ksoftirqd/107:40:111
1168999222,4cyclictest131rcu_preempt07:10:151
1168999222,16cyclictest3383-21ssh12:05:141
1168999222,16cyclictest3383-21ssh12:05:141
1165699225,8cyclictest131rcu_preempt12:15:020
11656992213,6cyclictest8879-21ntp_states11:30:270
11748992115,4cyclictest321rcuc/309:17:463
11748992114,4cyclictest321rcuc/310:27:223
11748992114,4cyclictest321rcuc/310:27:223
11748992113,5cyclictest0-21swapper/311:01:413
1173299219,9cyclictest15629-21ssh10:22:562
1173299219,8cyclictest27-21ksoftirqd/210:15:082
1173299218,5cyclictest8943-21kthreadcore12:25:212
1173299214,9cyclictest16963-21grep07:15:192
1173299213,14cyclictest14630-21fschecks_time08:20:152
1173299211,17cyclictest10332-21kthreadcore08:15:192
1173299211,16cyclictest14445-21kthreadcore10:40:222
1173299211,14cyclictest20716-21cut09:00:162
1173299211,12cyclictest15425-21kthreadcore12:10:212
1173299211,12cyclictest15425-21kthreadcore12:10:212
1173299211,11cyclictest30899-21grep10:30:302
11732992110,9cyclictest14073-21ssh11:35:062
11732992110,9cyclictest11172-21ssh12:09:022
11732992110,9cyclictest11172-21ssh12:09:022
1173299210,11cyclictest8615-21timerwakeupswit08:10:282
1168999218,6cyclictest131rcu_preempt07:30:171
1168999212,16cyclictest7511-21ssh10:55:181
1168999211,15cyclictest4513-21latency_hist08:10:011
1168999211,15cyclictest4513-21latency_hist08:10:001
1168999211,13cyclictest6888-21ssh09:15:131
11689992110,8cyclictest21-21ksoftirqd/111:35:131
1165699215,12cyclictest16009-21ssh11:00:140
1165699212,15cyclictest854-21sshd10:45:130
1165699211,15cyclictest18707-21sh10:05:150
1165699211,10cyclictest13656-21ps07:45:180
1174899203,13cyclictest36066irq/59-eth011:05:133
11748992015,3cyclictest0-21swapper/312:25:173
11748992015,3cyclictest0-21swapper/307:15:253
11748992014,4cyclictest321rcuc/309:38:383
11748992014,4cyclictest0-21swapper/312:10:233
11748992014,4cyclictest0-21swapper/312:10:223
11748992014,4cyclictest0-21swapper/312:05:123
11748992014,4cyclictest0-21swapper/312:05:123
11748992014,4cyclictest0-21swapper/311:56:413
11748992014,4cyclictest0-21swapper/311:40:273
11748992014,4cyclictest0-21swapper/309:30:133
11748992014,4cyclictest0-21swapper/309:12:303
11748992013,5cyclictest0-21swapper/311:47:503
11748992013,5cyclictest0-21swapper/310:03:483
11748992013,4cyclictest0-21swapper/309:20:563
11748992012,6cyclictest0-21swapper/309:49:413
11748992011,7cyclictest0-21swapper/310:06:073
11748992010,6cyclictest35966irq/58-eth009:25:133
1173299209,9cyclictest24730-21ssh10:07:002
1173299209,9cyclictest1-21systemd07:39:392
1173299202,10cyclictest8313-21/usr/sbin/munin07:40:262
1173299201,9cyclictest25358-21kthreadcore08:30:192
1173299201,9cyclictest13139-21kthreadcore07:10:212
1173299201,12cyclictest0-21swapper/209:35:132
1173299201,10cyclictest22453-21cut10:45:182
11732992010,7cyclictest16755-21kernelversion09:20:202
1168999202,11cyclictest3364-21ssh10:15:211
11689992012,5cyclictest0-21swapper/111:55:171
11689992010,7cyclictest19586-21awk11:00:251
1165699209,8cyclictest16067-21forks12:30:160
1165699206,10cyclictest12-21ksoftirqd/007:30:140
1165699203,12cyclictest5272-21ssh09:55:130
1165699202,15cyclictest6940-21ssh10:55:140
1165699201,16cyclictest21156-21ps08:25:210
11656992010,7cyclictest0-21swapper/008:45:020
1165699200,15cyclictest16881-21ls07:15:180
1174899192,12cyclictest35966irq/58-eth010:35:133
11748991914,3cyclictest321rcuc/308:15:253
11748991913,4cyclictest0-21swapper/310:10:033
11748991913,4cyclictest0-21swapper/309:50:273
11748991913,4cyclictest0-21swapper/308:55:273
11748991913,4cyclictest0-21swapper/308:40:193
11748991913,4cyclictest0-21swapper/308:25:223
11748991913,4cyclictest0-21swapper/307:40:263
11748991911,6cyclictest0-21swapper/312:15:353
11748991910,7cyclictest0-21swapper/311:53:013
1173299196,5cyclictest13291-21fschecks_count07:45:162
1173299191,9cyclictest7920-21kthreadcore09:15:182
1173299191,8cyclictest21753-21fschecks_time07:20:162
1173299191,4cyclictest0-21swapper/209:51:202
11732991912,5cyclictest27909-21ssh10:10:262
1173299191,15cyclictest24928-21kthreadcore07:55:232
1168999197,8cyclictest21-21ksoftirqd/108:30:141
1168999194,11cyclictest0-21swapper/112:20:131
1168999193,13cyclictest10761-21ssh09:35:151
1168999191,3cyclictest131rcu_preempt08:00:201
1168999191,15cyclictest24189-21ssh09:45:131
1168999191,15cyclictest10000-21unixbench_singl08:45:281
1168999191,14cyclictest30385-21ssh12:00:251
1168999191,13cyclictest2402-21/usr/sbin/munin09:10:301
1165699199,8cyclictest29037-21sed11:05:300
1165699193,14cyclictest15344-21ps07:10:280
11656991910,7cyclictest24654-21cron09:25:020
1174899182,10cyclictest0-21swapper/309:05:143
11748991812,4cyclictest0-21swapper/309:40:153
11748991812,4cyclictest0-21swapper/308:10:183
11748991812,4cyclictest0-21swapper/308:00:273
11748991812,4cyclictest0-21swapper/307:55:143
1174899181,11cyclictest0-21swapper/309:05:013
1168999189,6cyclictest19632-21ssh09:20:261
1168999185,11cyclictest20470-21kthreadcore07:50:211
1168999183,10cyclictest1549-21find11:10:121
1168999183,10cyclictest0-21swapper/110:10:051
1168999181,9cyclictest25724-21irqcore09:05:181
11689991810,5cyclictest7705-21kthreadcore12:25:201
1165699189,7cyclictest30884-21memory08:00:230
1165699188,8cyclictest32143-21cat12:20:210
1165699187,8cyclictest23193-21apt-get11:40:130
1165699186,9cyclictest12-21ksoftirqd/008:45:130
1165699186,4cyclictest16766systemd-network10:37:520
1165699186,3cyclictest131rcu_preempt11:25:180
1165699182,13cyclictest21434-21ssh10:25:270
1165699182,13cyclictest21434-21ssh10:25:270
1165699182,11cyclictest10533-21ssh11:15:110
1165699182,10cyclictest0-21swapper/008:50:140
1165699181,3cyclictest0-21swapper/011:55:440
11656991811,5cyclictest12646-21idleruntime12:10:160
1165699181,14cyclictest6793-21kthreadcore08:10:210
11656991810,5cyclictest9126-21tail10:20:160
1174899177,6cyclictest321rcuc/308:50:143
1174899172,9cyclictest0-21swapper/308:20:143
11748991711,4cyclictest0-21swapper/308:31:263
1174899171,13cyclictest0-21swapper/308:35:203
1174899171,12cyclictest0-21swapper/308:45:153
1173299171,6cyclictest2993266diskmemload10:00:152
1168999179,3cyclictest21764-21hwlatdetect07:20:171
1168999178,7cyclictest19507-21perl08:25:171
1168999173,11cyclictest24947-21kthreadcore07:55:231
1168999173,10cyclictest0-21swapper/108:20:131
1168999171,7cyclictest23249-21ntp_states11:20:251
1168999171,7cyclictest23249-21ntp_states11:20:251
1168999171,6cyclictest2133-21kthreadcore08:40:201
11689991713,2cyclictest9650irq/52-mmc010:01:191
11689991712,3cyclictest0-21swapper/107:25:241
1168999171,14cyclictest11723-21latency_hist07:10:011
1168999171,13cyclictest0-21swapper/111:15:131
1165699179,5cyclictest19278-21kthreadcore07:50:190
1165699175,9cyclictest26615-21grep07:55:310
1165699173,11cyclictest9222-21cut09:15:210
1165699173,11cyclictest681-21kthreadcore10:50:220
1165699173,11cyclictest0-21swapper/008:15:130
1165699172,11cyclictest11713-21cat07:10:020
1165699171,5cyclictest53646usb-storage07:20:040
11656991711,4cyclictest1073-21munin-plugin-st08:40:010
1165699171,13cyclictest11000-21kthreadcore11:50:220
1165699170,2cyclictest131rcu_preempt09:52:440
1174899162,5cyclictest0-21swapper/307:10:143
11748991610,4cyclictest0-21swapper/307:50:233
11748991610,4cyclictest0-21swapper/307:30:243
11748991610,4cyclictest0-21swapper/307:25:143
1168999169,5cyclictest8156-21/usr/sbin/munin08:10:231
1168999168,5cyclictest12635-21kthreadcore08:15:211
1168999167,7cyclictest30911-21cut08:35:181
1168999163,10cyclictest4360-21kthreadcore07:35:171
11689991611,3cyclictest24273-21timerandwakeup09:00:271
11689991611,3cyclictest0-21swapper/109:53:511
11689991611,3cyclictest0-21swapper/109:44:261
1165699168,6cyclictest1425-21cat09:10:240
1165699164,9cyclictest8465-21fschecks_count07:40:130
1165699164,9cyclictest27294-21kthreadcore08:30:220
1165699162,10cyclictest0-21swapper/008:55:140
1165699162,10cyclictest0-21swapper/008:20:140
11656991611,3cyclictest0-21swapper/010:10:050
1165699161,12cyclictest3057-21kthreadcore08:05:230
1165699161,12cyclictest3057-21kthreadcore08:05:230
1165699161,12cyclictest29960-21wc09:10:020
1165699161,11cyclictest22785-21tr09:00:210
1165699161,11cyclictest21645-21sshd09:42:260
1173299159,4cyclictest11537-21apt-get07:10:022
1165699152,10cyclictest13901-21latency_hist10:00:010
1165699151,11cyclictest26455-21diskstats07:25:170
1174899146,6cyclictest321rcuc/307:35:123
1174899111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional