You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-11 - 23:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Fri Sep 11, 2026 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5952996734,19cyclictest35966irq/58-eth011:20:133
593599636,9cyclictest131rcu_preempt08:40:141
5935996147,7cyclictest10004-21sh09:20:121
593599598,7cyclictest131rcu_preempt10:35:141
593599594,42cyclictest1192-21ssh10:45:111
5935995912,8cyclictest131rcu_preempt07:30:121
593499545,38cyclictest6684-21ssh12:00:120
593499544,34cyclictest24625-21ssh10:40:130
593499544,34cyclictest24625-21ssh10:40:130
594399521,46cyclictest7318-21apt-get07:45:112
593499511,42cyclictest10509-21ssh12:20:130
594399491,43cyclictest0-21swapper/211:25:132
593599493,10cyclictest131rcu_preempt07:35:141
594399481,43cyclictest0-21swapper/212:35:112
594399473,2cyclictest17526-21apt-get11:30:192
594399472,40cyclictest0-21swapper/208:15:122
593599476,23cyclictest17529-21ssh11:30:191
593499473,31cyclictest26425-21ssh11:35:130
594399433,6cyclictest0-21swapper/209:05:132
594399431,4cyclictest27346irq/40-dwc2_hso09:50:182
594399431,3cyclictest19263-21ssh10:55:132
594399431,3cyclictest0-21swapper/209:10:122
594399431,2cyclictest32586-21apt-get08:45:122
594399431,10cyclictest0-21swapper/209:40:142
594399430,39cyclictest15625-21ssh10:35:132
594399421,37cyclictest0-21swapper/208:40:132
593599411,30cyclictest23988-21diskmemload09:45:131
593599411,30cyclictest23988-21diskmemload09:45:121
594399402,33cyclictest0-21swapper/211:49:282
594399401,7cyclictest6875-21ssh11:23:372
594399401,3cyclictest0-21swapper/210:40:132
594399401,3cyclictest0-21swapper/210:40:122
594399401,33cyclictest0-21swapper/210:50:132
594399400,4cyclictest1188-21apt-get10:45:122
593599407,8cyclictest131rcu_preempt11:40:131
593599407,8cyclictest131rcu_preempt11:40:131
593599401,31cyclictest11978-21ssh10:15:121
594399391,4cyclictest22976-21apt-get08:35:142
594399391,4cyclictest0-21swapper/207:40:132
594399391,3cyclictest0-21swapper/211:35:142
593599392,25cyclictest19027-21ssh09:25:131
594399381,7cyclictest18528-21irqcore08:30:182
594399381,6cyclictest11871-21ssh10:15:042
594399381,4cyclictest0-21swapper/210:05:132
594399381,4cyclictest0-21swapper/207:25:132
594399381,33cyclictest1176-21ssh12:15:022
5935993818,16cyclictest31736-21ssh09:50:181
5952993721,9cyclictest35966irq/58-eth009:30:143
594399373,2cyclictest0-21swapper/212:05:122
594399373,11cyclictest0-21swapper/212:20:132
594399371,8cyclictest23896-21tr10:20:222
594399371,8cyclictest21928-21ssh09:25:212
594399371,6cyclictest6129-21tr10:10:272
594399371,4cyclictest31873-21kthreadcore12:30:282
594399371,2cyclictest13839-21apt-get11:10:112
594399370,2cyclictest22660-21apt-get09:45:142
594399370,2cyclictest22660-21apt-get09:45:142
5934993725,8cyclictest12080-21apt-get07:50:120
595299364,23cyclictest35966irq/58-eth012:25:113
594399361,7cyclictest5397-21wc12:15:292
594399361,7cyclictest21837-21fschecks_count08:00:152
593599367,22cyclictest23988-21diskmemload10:50:141
595299350,22cyclictest35966irq/58-eth009:20:123
594399352,4cyclictest0-21swapper/208:50:132
594399351,8cyclictest20376-21proc_pri07:55:272
594399351,8cyclictest20376-21proc_pri07:55:262
594399351,7cyclictest25989-21latency_hist10:05:022
594399351,7cyclictest19875-21rm12:25:152
594399351,6cyclictest8289-21ssh09:35:252
594399351,5cyclictest11728-21latency_hist11:45:022
594399351,5cyclictest11728-21latency_hist11:45:022
594399351,32cyclictest4185-21ssh09:35:002
594399351,31cyclictest9778-21/usr/sbin/munin08:55:112
594399350,5cyclictest16936-21cut10:00:022
593599357,6cyclictest131rcu_preempt11:10:121
594399341,7cyclictest27346irq/40-dwc2_hso07:50:122
594399341,6cyclictest22758-21acpi11:15:102
594399341,5cyclictest531-21kthreadcore07:35:212
594399340,5cyclictest13800-21cut08:25:182
594399340,5cyclictest13800-21cut08:25:182
593599345,18cyclictest31838-21ssh11:20:121
593499341,30cyclictest13702-21ssh09:40:120
594399331,6cyclictest798-21sh11:00:352
594399331,5cyclictest29938-21ssh11:55:042
594399331,5cyclictest29626-21basename11:55:022
594399331,4cyclictest1552-21grep10:25:282
594399331,4cyclictest13922-21ls09:20:292
594399331,10cyclictest8556-21ntp_states11:05:262
594399330,6cyclictest19226-21ntpq07:20:252
593599334,20cyclictest8419-21ssh11:25:121
5935993314,9cyclictest131rcu_preempt08:10:121
593499332,25cyclictest26291-21ssh10:05:120
594399324,3cyclictest8463-21cut10:30:212
594399322,3cyclictest1100-21cut09:15:162
594399321,8cyclictest2555-21grep08:10:262
594399321,8cyclictest11604-21kthreadcore08:20:222
594399321,7cyclictest30429-21grep08:05:292
594399321,7cyclictest17827-21kthreadcore09:00:232
594399320,4cyclictest10961-21apt-get07:15:132
593499323,9cyclictest8203-21ntp_192.168.11409:35:240
593499321,21cyclictest25024-21kworker/u8:212:25:120
5952993113,12cyclictest35966irq/58-eth010:10:173
594399311,3cyclictest0-21swapper/212:00:042
594399310,28cyclictest6157-21grep07:10:152
593599313,22cyclictest30080-21ssh11:55:131
593599312,6cyclictest131rcu_preempt12:10:191
593599312,18cyclictest6674-21ssh12:00:121
5935993120,4cyclictest211ksoftirqd/112:35:121
594399301,7cyclictest25325-21/usr/sbin/munin07:30:232
593599301,19cyclictest6630-21ssh10:30:121
593499302,22cyclictest5074-21kthreadcore10:10:250
593499302,22cyclictest19231-21ssh11:48:260
593499301,26cyclictest3047-21sh11:40:140
593499301,26cyclictest3047-21sh11:40:140
595299294,17cyclictest35966irq/58-eth011:45:113
593599292,17cyclictest19278-21ssh10:55:141
5934992923,3cyclictest31724-21apt-get09:50:180
593499292,23cyclictest13871-21ssh11:10:130
5934992916,10cyclictest19260-21apt-get10:55:130
595299283,12cyclictest35966irq/58-eth011:00:143
593599282,21cyclictest9669-21ssh12:19:321
593599282,18cyclictest904-21ssh09:15:121
593599281,21cyclictest0-21swapper/112:05:121
593499280,6cyclictest19216-21cut07:55:220
593499280,6cyclictest19216-21cut07:55:210
595299273,17cyclictest35966irq/58-eth010:50:143
595299273,15cyclictest0-21swapper/312:19:323
593599272,18cyclictest19571-21ssh12:25:131
593499274,9cyclictest19544-21sed09:25:180
5934992721,3cyclictest5168-21apt-get12:35:120
5934992719,5cyclictest13430-21apt-get08:25:130
5934992719,5cyclictest13430-21apt-get08:25:130
5934992718,7cyclictest15614-21apt-get10:35:150
593499271,22cyclictest6639-21sh10:30:140
593499271,20cyclictest31799-21/usr/sbin/munin11:20:250
593499271,10cyclictest24855-21ssh10:03:480
593499270,10cyclictest29876-21latency_hist10:25:020
595299262,17cyclictest35966irq/58-eth010:30:133
595299261,17cyclictest35966irq/58-eth009:55:123
593599263,15cyclictest21-21ksoftirqd/110:10:171
593599262,7cyclictest131rcu_preempt11:46:261
593599262,7cyclictest131rcu_preempt10:04:241
593599261,19cyclictest28694-21ssh12:30:181
593499269,12cyclictest12-21ksoftirqd/010:53:250
593499263,18cyclictest11959-21apt-get10:15:090
5934992619,3cyclictest21687-21apt-get08:00:120
5934992617,6cyclictest20561-21apt-get07:25:120
593499261,21cyclictest0-21swapper/009:55:120
595299251,15cyclictest35966irq/58-eth011:35:123
593599252,16cyclictest24389-21ssh09:10:131
593499257,5cyclictest0-21swapper/008:45:120
5934992518,4cyclictest892-21apt-get09:15:130
593499251,7cyclictest2344-21ntp_192.168.11408:10:240
5934992516,7cyclictest28298-21ssh11:00:150
595299247,11cyclictest0-21swapper/309:53:263
594399241,3cyclictest0-21swapper/207:10:022
593599245,10cyclictest5933-21cyclictest11:50:141
5935992417,4cyclictest930-21snmpd12:23:381
593499245,10cyclictest0-21swapper/011:06:000
593499244,6cyclictest12-21ksoftirqd/008:40:230
593499244,5cyclictest8610-21ls08:20:030
593499242,6cyclictest11068-21tr11:25:200
5934992413,7cyclictest19405-21apt-get09:05:130
593499241,18cyclictest23988-21diskmemload09:20:130
5934992410,3cyclictest131rcu_preempt12:30:180
593499240,9cyclictest5676-21usage_Anybus10:45:290
595299233,14cyclictest0-21swapper/311:44:283
595299233,14cyclictest0-21swapper/311:44:273
595299233,13cyclictest0-21swapper/309:39:043
593599231,18cyclictest21925-21ls08:00:161
593499235,10cyclictest0-21swapper/011:33:020
593499234,3cyclictest0-21swapper/011:15:120
593499233,12cyclictest0-21swapper/008:35:120
593499232,16cyclictest20997-21ssh11:50:130
5934992318,2cyclictest30148-21apt-get07:35:140
593499231,17cyclictest28046-21ssh09:30:150
593499231,17cyclictest19485-21ssh12:05:250
593499230,8cyclictest0-21swapper/010:25:180
595299229,9cyclictest35966irq/58-eth010:55:143
595299229,10cyclictest0-21swapper/310:25:253
595299223,12cyclictest0-21swapper/310:05:243
5935992216,4cyclictest0-21swapper/107:10:021
593599221,15cyclictest12262-21cut07:50:151
593499227,4cyclictest0-21swapper/012:13:160
593499227,3cyclictest0-21swapper/011:56:560
593499227,3cyclictest0-21swapper/009:45:260
593499227,3cyclictest0-21swapper/009:45:250
593499226,4cyclictest12-21ksoftirqd/007:45:260
593499226,3cyclictest0-21swapper/008:05:030
593499224,4cyclictest12-21ksoftirqd/012:17:170
593499221,6cyclictest0-21swapper/008:30:300
593499221,18cyclictest24671-21ls09:10:170
593499221,13cyclictest2498-21find07:40:120
595299219,9cyclictest0-21swapper/311:55:263
595299213,12cyclictest0-21swapper/312:20:133
595299212,10cyclictest35966irq/58-eth007:55:153
595299212,10cyclictest35966irq/58-eth007:55:143
5952992115,4cyclictest0-21swapper/311:05:243
5952992113,4cyclictest0-21swapper/312:35:133
595299211,16cyclictest35966irq/58-eth011:10:123
595299211,13cyclictest0-21swapper/309:10:143
593599217,4cyclictest0-21swapper/110:05:171
593599212,14cyclictest30549-21kworker/u8:109:59:091
593599211,3cyclictest131rcu_preempt11:15:151
593599211,3cyclictest131rcu_preempt10:25:131
593599211,15cyclictest21042-21ssh10:20:131
593499215,3cyclictest0-21swapper/008:20:050
5934992112,4cyclictest0-21swapper/007:10:130
593499211,16cyclictest8674-21ls08:50:270
595299209,8cyclictest0-21swapper/309:40:523
5952992014,4cyclictest0-21swapper/310:35:213
5952992014,4cyclictest0-21swapper/310:00:293
5952992011,4cyclictest0-21swapper/309:27:303
595299201,13cyclictest0-21swapper/312:05:123
5952992010,7cyclictest0-21swapper/312:30:333
5952992010,7cyclictest0-21swapper/310:15:033
593599204,7cyclictest0-21swapper/108:45:171
593599201,17cyclictest28513-21cut11:00:171
593499208,4cyclictest0-21swapper/007:20:130
595299193,11cyclictest0-21swapper/310:48:263
595299193,10cyclictest0-21swapper/307:25:143
595299192,11cyclictest0-21swapper/307:50:123
5952991913,4cyclictest0-21swapper/311:54:473
5952991913,4cyclictest0-21swapper/310:43:203
5952991913,4cyclictest0-21swapper/310:43:203
5952991913,4cyclictest0-21swapper/310:20:163
5952991913,4cyclictest0-21swapper/309:45:133
5952991913,4cyclictest0-21swapper/309:45:123
5952991912,5cyclictest0-21swapper/312:14:063
593599198,6cyclictest31075-21kthreadcore09:30:251
593599191,14cyclictest30023-21ssh11:35:241
593599191,13cyclictest7373-21kthreadcore07:10:201
5934991913,3cyclictest25343-21apt-get07:30:130
595299189,5cyclictest321rcuc/311:15:113
595299182,11cyclictest0-21swapper/312:00:143
595299182,10cyclictest0-21swapper/309:05:133
595299182,10cyclictest0-21swapper/308:50:133
595299182,10cyclictest0-21swapper/308:35:143
595299182,10cyclictest0-21swapper/308:25:143
595299182,10cyclictest0-21swapper/308:25:133
595299182,10cyclictest0-21swapper/307:30:143
5952991812,4cyclictest0-21swapper/311:30:383
5952991812,4cyclictest0-21swapper/311:25:363
5952991812,4cyclictest0-21swapper/309:16:573
595299181,13cyclictest0-21swapper/307:40:023
595299181,11cyclictest0-21swapper/307:45:133
593599189,6cyclictest8880-21wc08:20:171
593599189,6cyclictest18444-21grep09:00:291
593599188,6cyclictest0-21swapper/107:45:111
593599186,9cyclictest0-21swapper/110:40:051
593599186,9cyclictest0-21swapper/110:40:041
593599186,3cyclictest0-21swapper/109:39:261
593599185,10cyclictest26698-21fschecks_count08:05:161
593599184,8cyclictest201rcuc/111:05:111
593599184,4cyclictest21-21ksoftirqd/108:35:281
593599182,13cyclictest15830-21kthreadcore09:40:211
593599181,3cyclictest131rcu_preempt08:55:111
5935991812,4cyclictest0-21swapper/107:40:111
593599180,11cyclictest4934-21find08:50:111
593499188,6cyclictest9876-21df08:55:120
593499180,3cyclictest91-21pr/ttyS009:05:020
595299172,10cyclictest0-21swapper/308:55:123
595299172,10cyclictest0-21swapper/308:45:123
595299172,10cyclictest0-21swapper/308:15:123
595299172,10cyclictest0-21swapper/308:05:033
595299171,12cyclictest0-21swapper/307:40:263
593599178,6cyclictest23995-21munin-plugin-st09:10:021
593599172,5cyclictest7331-21mailstats08:15:271
593599171,9cyclictest131rcu_preempt07:25:191
593599171,8cyclictest13139-21tr07:15:201
593599171,6cyclictest22807-21latency_hist08:35:011
593599171,13cyclictest17264-21kthreadcore07:20:191
593499171,11cyclictest0-21swapper/007:15:120
595299167,6cyclictest0-21swapper/308:20:233
595299162,10cyclictest0-21swapper/309:00:153
595299161,12cyclictest0-21swapper/308:40:173
595299161,10cyclictest0-21swapper/307:20:013
5952991610,4cyclictest0-21swapper/308:00:253
593599168,6cyclictest13480-21awk08:25:141
593599168,6cyclictest13480-21awk08:25:131
593599162,10cyclictest0-21swapper/107:55:151
593599162,10cyclictest0-21swapper/107:55:151
595299159,4cyclictest33-21ksoftirqd/307:20:153
595299158,5cyclictest0-21swapper/307:10:253
595299158,4cyclictest0-21swapper/308:10:243
593499141,11cyclictest5933-21cyclictest07:10:010
595299137,4cyclictest0-21swapper/308:30:183
595299101,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional