You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-07 - 01:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot3s.osadl.org (updated Mon Jul 06, 2026 12:45:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
836912998818,61cyclictest16850irq/134-800000011:54:400
836912998530,51cyclictest866263-21ssh09:58:000
836912998530,51cyclictest866263-21ssh09:58:000
836912998421,58cyclictest905359-21ssh11:19:540
836912998327,40cyclictest932585-21ssh12:16:500
836923998110,54cyclictest907877-21ssh11:25:012
836912998033,43cyclictest16950irq/224-800000008:00:010
836912997935,40cyclictest16950irq/224-800000010:19:160
83691299792,60cyclictest845257-21ssh09:14:310
836912997923,52cyclictest893102-21ssh10:54:140
836912997917,13cyclictest886497-21ssh10:40:030
836912997830,42cyclictest895869-21ssh10:59:590
836912997822,52cyclictest905793-21ssh11:20:200
83691299781,61cyclictest846082-21ssh09:15:550
83692799763,63cyclictest120-21kswapd011:15:023
836912997618,48cyclictest16850irq/134-800000012:33:080
836912997526,45cyclictest929368-21latency_hist12:10:020
836912997526,45cyclictest929368-21latency_hist12:10:010
83691299752,56cyclictest925403-21ssh12:01:380
836912997517,53cyclictest16850irq/134-800000011:25:350
836912997517,12cyclictest930509-21ssh12:12:290
836912997429,40cyclictest938729-21ssh12:30:000
836912997429,40cyclictest938729-21ssh12:30:000
836912997426,31cyclictest899621-21ssh11:07:430
836912997420,12cyclictest941678-21ssh12:35:430
83691299741,69cyclictest862948-21ssh09:50:540
83691299741,57cyclictest0-21swapper/011:43:380
836912997327,42cyclictest900938-21ssh11:10:130
83691299731,68cyclictest0-21swapper/009:22:520
83691299731,68cyclictest0-21swapper/009:22:510
836912997315,53cyclictest16850irq/134-800000011:04:430
836923997025,40cyclictest30-21ksoftirqd/209:18:552
83691299702,63cyclictest2639831-1kworker/0:2H08:10:060
83691299702,63cyclictest2639831-1kworker/0:2H08:10:060
836912997026,27cyclictest914106-21ssh11:38:070
836912997026,26cyclictest887133-21ssh10:41:290
83692399692,12cyclictest865499-21sshd09:56:172
83692399692,12cyclictest865499-21sshd09:56:162
836912996918,41cyclictest16850irq/134-800000010:34:320
836912996916,6cyclictest16850irq/134-800000009:47:070
83691299691,2cyclictest0-21swapper/011:47:360
836923996826,37cyclictest877335-21sh10:20:472
83691299686,6cyclictest16850irq/134-800000010:45:120
836912996825,39cyclictest364-21snmpd08:42:490
836912996820,44cyclictest16850irq/134-800000009:26:170
836912996819,44cyclictest16850irq/134-800000010:08:040
836912996819,44cyclictest16850irq/134-800000010:08:040
836912996728,23cyclictest880494-21ssh10:27:390
836912996721,2cyclictest0-21swapper/012:23:160
836912996718,44cyclictest16850irq/134-800000009:33:190
836912996712,14cyclictest275-21dbus-daemon08:55:000
83692399662,13cyclictest924333-21ssh11:59:462
83692399662,13cyclictest924333-21ssh11:59:462
83692399662,13cyclictest917337-21munin-run11:44:592
83691299663,20cyclictest912560-21rm11:35:010
836912996621,41cyclictest16950irq/224-800000012:00:010
836912996621,41cyclictest16950irq/224-800000012:00:010
83691299661,60cyclictest0-21swapper/009:38:140
836912996610,45cyclictest16850irq/134-800000009:41:230
83692399642,15cyclictest888674-21munin-run10:45:012
836923996421,39cyclictest844069-21sh09:11:412
836912996459,3cyclictest0-21swapper/010:10:030
83691299643,20cyclictest879140-21munin-run10:25:000
83691299643,14cyclictest837413-21munin-run07:20:020
836912996411,5cyclictest16850irq/134-800000010:03:270
83692399631,12cyclictest882866-21ssh10:32:372
83692799621,14cyclictest3467602-21systemd-journal11:57:283
83692799621,14cyclictest3467602-21systemd-journal11:57:273
83692399622,15cyclictest941119-21ssh12:35:002
83692399622,13cyclictest929578-21sh12:10:242
83692399621,11cyclictest48-21kcompactd011:25:052
83691299623,14cyclictest842069-21sshd08:45:290
836923996133,22cyclictest30-21ksoftirqd/209:48:512
83692399612,13cyclictest840226-21munin-run08:15:002
83692399612,13cyclictest840226-21munin-run08:14:592
83692399612,13cyclictest838700-21munin-run07:45:012
83692399612,12cyclictest850322-21munin-run09:25:022
83692399612,12cyclictest850322-21munin-run09:25:022
836923996118,39cyclictest868410-21sh10:02:262
836923996118,39cyclictest30-21ksoftirqd/211:47:512
836923996110,4cyclictest0-21swapper/210:28:412
836912996118,4cyclictest839188-21latency_hist07:55:020
83691299611,55cyclictest0-21swapper/008:58:170
836912996112,6cyclictest16850irq/134-800000008:03:170
836927996010,11cyclictest37-21ksoftirqd/312:16:063
836923996018,37cyclictest904669-21sh11:18:122
83692399601,13cyclictest48-21kcompactd012:05:122
83692399601,13cyclictest48-21kcompactd012:05:122
836912996012,7cyclictest841002-21cron08:30:010
836927995938,14cyclictest3467602-21systemd-journal11:00:003
83692399593,7cyclictest936700-21ssh12:25:152
83692399593,7cyclictest936700-21ssh12:25:152
83692399593,15cyclictest920819-21ssh11:52:072
83692399592,12cyclictest857077-21ssh09:39:042
836923995917,37cyclictest843139-21diskmemload11:08:332
836923995912,4cyclictest941802-21ssh12:35:592
83692399591,13cyclictest838919-21sshd07:48:162
83692399591,13cyclictest364-21snmpd09:40:322
83692399591,13cyclictest364-21snmpd08:17:102
83692399591,13cyclictest364-21snmpd08:17:102
83692399591,12cyclictest852861-21latency_hist09:30:022
836912995917,4cyclictest840748-21kworker/u8:009:10:020
83691299591,3cyclictest0-21swapper/008:20:220
83692799586,9cyclictest37-21ksoftirqd/311:40:003
83692799585,11cyclictest37-21ksoftirqd/310:10:263
83692799582,13cyclictest37-21ksoftirqd/312:22:533
836927995818,9cyclictest37-21ksoftirqd/310:44:173
836927995816,10cyclictest37-21ksoftirqd/311:02:573
836923995816,38cyclictest932395-1kworker/u9:112:22:042
836923995814,11cyclictest918991-1kworker/u9:212:01:462
83692399581,15cyclictest364-21snmpd07:23:282
83692399581,10cyclictest932321-21sshd12:16:182
83692799577,6cyclictest37-21ksoftirqd/309:30:013
83692799576,10cyclictest37-21ksoftirqd/311:35:013
83692799575,7cyclictest37-21ksoftirqd/312:30:013
83692799575,7cyclictest37-21ksoftirqd/312:30:013
836927995745,8cyclictest37-21ksoftirqd/309:50:003
836927995710,9cyclictest37-21ksoftirqd/312:05:023
836923995725,27cyclictest30-21ksoftirqd/211:03:362
83692399572,14cyclictest893440-21ssh10:54:562
83692399572,13cyclictest872650-21ssh10:11:022
836923995714,10cyclictest907446-1kworker/u9:111:32:212
83692399571,11cyclictest871935-21munin-run10:10:012
83692399571,11cyclictest871935-21munin-run10:10:012
83691299577,11cyclictest275-21dbus-daemon08:40:010
83691299577,11cyclictest275-21dbus-daemon08:40:010
83692799565,28cyclictest840213-21kworker/u8:208:30:023
836927995621,11cyclictest37-21ksoftirqd/309:20:023
836927995621,11cyclictest37-21ksoftirqd/309:20:023
836927995612,7cyclictest37-21ksoftirqd/308:40:023
836927995612,7cyclictest37-21ksoftirqd/308:40:023
83692399564,6cyclictest131rcu_preempt09:00:002
83692399562,12cyclictest364-21snmpd09:54:452
836923995618,34cyclictest885743-1kworker/u9:210:39:532
83692399561,19cyclictest891096-21cron10:50:012
83692399561,12cyclictest893883-21ssh10:55:262
83691299561,51cyclictest842916-21date09:05:020
83692799558,18cyclictest37-21ksoftirqd/312:33:493
83692799556,8cyclictest37-21ksoftirqd/310:20:443
83692799555,9cyclictest37-21ksoftirqd/310:32:553
83692799555,9cyclictest37-21ksoftirqd/310:10:003
83692799555,9cyclictest37-21ksoftirqd/310:09:593
83692399552,14cyclictest854439-21ssh09:33:232
83692399551,14cyclictest914504-21ssh11:39:062
83692399551,13cyclictest874597-21ssh10:15:072
83692399551,12cyclictest839863-21latency_hist08:05:012
836912995549,3cyclictest0-21swapper/007:30:030
83691299553,15cyclictest3467602-21systemd-journal07:40:010
83691299553,15cyclictest3467602-21systemd-journal07:40:000
836912995515,4cyclictest3467602-21systemd-journal07:12:310
83692799548,10cyclictest37-21ksoftirqd/312:38:003
83692799546,8cyclictest37-21ksoftirqd/311:28:233
836927995435,14cyclictest275-21dbus-daemon08:00:003
83692399542,16cyclictest837912-21systemd-run07:30:012
836923995421,29cyclictest30-21ksoftirqd/211:13:262
83692399541,11cyclictest841494-21t-helper08:37:152
83692399541,11cyclictest841494-21t-helper08:37:142
836912995422,27cyclictest838936-21latency_hist07:50:010
83692799537,9cyclictest37-21ksoftirqd/310:02:453
83692799537,13cyclictest37-21ksoftirqd/310:54:543
83692799536,9cyclictest37-21ksoftirqd/312:09:413
83692799536,9cyclictest37-21ksoftirqd/312:09:413
83692799536,13cyclictest37-21ksoftirqd/310:19:573
83692799533,10cyclictest37-21ksoftirqd/310:35:303
83692799531,8cyclictest864757-21unin-run09:55:013
836927995312,12cyclictest37-21ksoftirqd/311:50:023
83692399532,13cyclictest364-21snmpd07:51:512
836923995316,33cyclictest840762-21latency_hist08:25:022
83692399531,13cyclictest843119-21sshd09:09:172
836923995310,4cyclictest0-21swapper/208:52:232
836917995335,12cyclictest1-21systemd12:15:001
836917995334,11cyclictest3467602-21systemd-journal08:40:011
836917995334,11cyclictest3467602-21systemd-journal08:40:001
83692799528,11cyclictest37-21ksoftirqd/311:16:003
83692799526,8cyclictest37-21ksoftirqd/312:10:233
83692799526,8cyclictest37-21ksoftirqd/311:40:483
83692799525,13cyclictest37-21ksoftirqd/311:55:033
83692799523,10cyclictest37-21ksoftirqd/308:55:023
83692799521,10cyclictest37-21ksoftirqd/310:27:463
836923995233,14cyclictest1-21systemd08:10:012
836923995233,14cyclictest1-21systemd08:10:012
83692399522,11cyclictest364-21snmpd09:02:472
83692399521,15cyclictest841234-21sshd08:30:262
836917995239,8cyclictest905400-21munin-plugin-st11:19:591
83691299522,19cyclictest840493-21munin-run08:20:000
83691299522,19cyclictest840493-21munin-run08:19:590
83692799518,10cyclictest37-21ksoftirqd/309:59:453
83692799518,10cyclictest37-21ksoftirqd/309:59:443
83692799516,7cyclictest37-21ksoftirqd/311:05:183
83692799512,9cyclictest37-21ksoftirqd/309:40:103
83692799511,8cyclictest37-21ksoftirqd/309:18:003
83692399512,12cyclictest364-21snmpd07:16:202
83692399511,12cyclictest364-21snmpd08:45:472
836917995135,11cyclictest23-21ksoftirqd/109:45:031
836912995123,25cyclictest12-21ksoftirqd/007:45:020
83692799506,9cyclictest37-21ksoftirqd/308:20:023
83692799506,9cyclictest37-21ksoftirqd/308:20:023
83692799501,5cyclictest0-21swapper/307:30:013
83692799501,16cyclictest37-21ksoftirqd/309:10:053
83692799501,12cyclictest37-21ksoftirqd/309:35:023
83692399501,11cyclictest364-21snmpd08:42:492
836912995015,9cyclictest12-21ksoftirqd/007:24:340
83692799499,5cyclictest131rcu_preempt10:46:413
83692799495,4cyclictest131rcu_preempt11:21:393
836923994931,12cyclictest275-21dbus-daemon08:30:002
83692399492,7cyclictest839430-21cron08:00:002
83692399492,18cyclictest838296-21wc07:35:022
836917994924,6cyclictest23-21ksoftirqd/111:36:091
83691299494,11cyclictest840246-21cat08:15:020
83691299494,11cyclictest840246-21cat08:15:010
836912994934,12cyclictest12-21ksoftirqd/007:30:030
836927994829,15cyclictest275-21dbus-daemon07:45:013
836917994833,11cyclictest3467602-21systemd-journal08:35:011
836917994827,15cyclictest275-21dbus-daemon08:10:021
836917994827,15cyclictest275-21dbus-daemon08:10:011
836927994725,16cyclictest3467602-21systemd-journal08:50:003
83692799471,7cyclictest856143-21ssh09:36:483
836923994720,23cyclictest838374-21sshd07:35:182
836923994720,23cyclictest838374-21sshd07:35:172
836917994730,12cyclictest3467602-21systemd-journal12:08:171
836917994730,12cyclictest3467602-21systemd-journal12:08:171
836917994727,13cyclictest23-21ksoftirqd/109:50:001
836912994717,4cyclictest841303-21latency_hist08:35:010
83692799468,6cyclictest37-21ksoftirqd/307:55:023
83692799466,8cyclictest37-21ksoftirqd/307:35:023
83692799465,8cyclictest37-21ksoftirqd/309:10:013
83692799464,8cyclictest37-21ksoftirqd/307:25:003
83692399462,11cyclictest837124-21systemctl07:15:012
83691799464,4cyclictest131rcu_preempt12:24:181
836917994625,15cyclictest275-21dbus-daemon11:15:011
83691799453,3cyclictest131rcu_preempt09:22:061
83691799453,3cyclictest131rcu_preempt09:22:051
836917994527,10cyclictest836919-21cat07:10:011
83692799446,9cyclictest37-21ksoftirqd/308:25:023
83692399441,12cyclictest837032-21date07:10:012
836917994432,7cyclictest838415-21munin-plugin-st07:40:011
836917994432,7cyclictest838415-21munin-plugin-st07:40:011
836917994428,10cyclictest895895-21munin-plugin-st11:00:001
836917994426,14cyclictest275-21dbus-daemon12:05:001
836917994424,15cyclictest886259-21readlink10:40:011
836912994411,6cyclictest12-21ksoftirqd/007:10:020
83692799437,8cyclictest37-21ksoftirqd/308:05:013
83692799433,9cyclictest37-21ksoftirqd/308:35:003
836927994336,5cyclictest37-21ksoftirqd/309:05:023
836917994333,6cyclictest881549-21munin-plugin-st10:30:011
83691799432,3cyclictest131rcu_preempt12:16:261
83692799425,7cyclictest37-21ksoftirqd/307:50:023
836917994229,9cyclictest1-21systemd08:50:011
836917994228,8cyclictest309-21in:imuxsock10:50:461
836917994226,11cyclictest309-21in:imuxsock12:00:011
836917994226,11cyclictest309-21in:imuxsock12:00:011
83691799422,36cyclictest23-21ksoftirqd/109:20:011
836917994221,17cyclictest1-21systemd11:55:011
83691799421,8cyclictest0-21swapper/108:25:201
83692799417,10cyclictest37-21ksoftirqd/308:15:033
83692799417,10cyclictest37-21ksoftirqd/308:15:023
83692799411,5cyclictest0-21swapper/307:15:023
836917994130,8cyclictest1-21systemd09:35:011
836917994127,11cyclictest23-21ksoftirqd/111:09:301
836917994127,10cyclictest23-21ksoftirqd/109:53:531
83691799412,3cyclictest131rcu_preempt11:24:191
83691799411,3cyclictest131rcu_preempt11:46:241
83692799407,8cyclictest37-21ksoftirqd/307:10:023
83691799404,3cyclictest131rcu_preempt11:40:281
836917994028,10cyclictest23-21ksoftirqd/110:18:251
836917994027,7cyclictest309-21in:imuxsock10:15:011
836917994026,12cyclictest23-21ksoftirqd/111:04:301
836917994026,11cyclictest23-21ksoftirqd/110:21:581
836917994026,11cyclictest23-21ksoftirqd/109:11:411
836917994023,12cyclictest1-21systemd10:00:011
836917994023,12cyclictest1-21systemd10:00:011
836917994022,14cyclictest3467602-21systemd-journal08:45:001
836917994020,13cyclictest309-21in:imuxsock09:10:011
83692799391,9cyclictest37-21ksoftirqd/307:35:073
83692799391,9cyclictest37-21ksoftirqd/307:35:063
83691799399,5cyclictest131rcu_preempt09:26:031
83691799394,3cyclictest131rcu_preempt10:49:371
83691799393,26cyclictest857545-21sh09:39:591
836917993931,5cyclictest869586-21sh10:05:001
836917993929,7cyclictest842335-21sshd08:54:181
836917993928,9cyclictest23-21ksoftirqd/111:32:521
836917993926,10cyclictest23-21ksoftirqd/110:41:381
836917993924,12cyclictest23-21ksoftirqd/109:00:001
836927993826,8cyclictest3467602-21systemd-journal08:10:013
836927993826,8cyclictest3467602-21systemd-journal08:10:003
836917993827,9cyclictest23-21ksoftirqd/110:33:191
836917993827,8cyclictest837961-21wc07:30:021
836917993825,9cyclictest838799-21cat07:45:011
836917993823,11cyclictest23-21ksoftirqd/112:37:221
836917993821,12cyclictest275-21dbus-daemon09:05:011
83692799375,10cyclictest37-21ksoftirqd/308:41:153
836927993727,6cyclictest842603-21sh08:59:593
83691799373,4cyclictest131rcu_preempt12:30:471
83691799373,3cyclictest131rcu_preempt11:25:131
836917993726,8cyclictest23-21ksoftirqd/112:25:221
836917993726,8cyclictest23-21ksoftirqd/112:25:221
836917993726,8cyclictest23-21ksoftirqd/107:50:011
836917993725,9cyclictest839425-21sshd07:59:171
83692799364,8cyclictest37-21ksoftirqd/307:17:333
836917993424,8cyclictest23-21ksoftirqd/110:05:551
836917993424,8cyclictest23-21ksoftirqd/110:05:551
836917993326,4cyclictest840766-21/usr/sbin/munin08:25:001
836917993323,7cyclictest23-21ksoftirqd/108:20:011
836917993323,7cyclictest23-21ksoftirqd/108:20:011
836917993323,7cyclictest23-21ksoftirqd/107:20:001
836917993317,12cyclictest275-21dbus-daemon07:25:011
836917993316,14cyclictest23-21ksoftirqd/107:15:021
836917993124,4cyclictest1-21systemd08:05:011
83691799312,3cyclictest131rcu_preempt08:15:001
83691799312,3cyclictest131rcu_preempt08:15:001
836917993021,7cyclictest23-21ksoftirqd/107:55:011
836917993017,10cyclictest23-21ksoftirqd/107:35:011
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional