You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-12 - 05:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Fri Jun 12, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
168599612,7cyclictest0-21swapper/219:15:132
168599573,36cyclictest18382-21ssh21:45:102
168599555,46cyclictest0-21swapper/219:35:112
1666995412,34cyclictest10728-21ssh23:10:130
168599533,4cyclictest0-21swapper/222:45:132
168599531,4cyclictest0-21swapper/219:10:122
166699532,41cyclictest15967-21ssh22:55:130
168599525,22cyclictest8042-21ssh22:15:132
168599511,4cyclictest0-21swapper/222:25:182
168599511,4cyclictest0-21swapper/222:25:182
167699512,2cyclictest131rcu_preempt21:10:131
167699481,6cyclictest131rcu_preempt22:20:131
166699485,30cyclictest27329-21ssh23:55:130
1666994831,11cyclictest23638-21ssh23:35:130
168599475,38cyclictest27346irq/40-dwc2_hso00:15:122
167699473,11cyclictest131rcu_preempt21:45:111
167699471,5cyclictest131rcu_preempt00:15:121
168599441,10cyclictest19661-21diskmemload23:50:122
168599440,7cyclictest0-21swapper/220:05:132
1676994420,12cyclictest21-21ksoftirqd/121:55:121
169399435,20cyclictest35966irq/58-eth021:40:123
168599423,6cyclictest522-21apt-get20:50:122
168599423,4cyclictest0-21swapper/221:00:112
1676994235,3cyclictest21-21ksoftirqd/123:15:141
167699423,28cyclictest2126-21ssh00:35:121
166699423,28cyclictest8048-21ssh22:15:130
166699422,26cyclictest25021-21ssh23:00:120
166699422,26cyclictest25021-21ssh23:00:120
169399414,26cyclictest35966irq/58-eth023:55:123
168599413,4cyclictest0-21swapper/222:05:132
1666994132,4cyclictest10207-21apt-get21:00:120
1666994121,15cyclictest28890-21ssh23:20:110
169399403,23cyclictest35966irq/58-eth023:25:113
168599401,8cyclictest14259-21kthreadcore22:35:222
1666994031,5cyclictest9001-21apt-get20:25:110
168599393,30cyclictest27346irq/40-dwc2_hso21:43:212
168599391,34cyclictest24097-21fschecks_time21:30:162
1676993919,12cyclictest0-21swapper/121:30:131
1666993924,9cyclictest14824-21apt-get21:25:140
168599386,29cyclictest23673-21switchtime23:15:302
168599382,7cyclictest0-21swapper/222:20:132
168599381,9cyclictest15276-21grep19:20:272
168599381,8cyclictest32396-21kthreadcore19:40:202
168599381,33cyclictest5880-21grep20:55:172
168599380,9cyclictest5915-21kthreadcore00:00:222
167699381,28cyclictest23629-21ssh23:35:121
1666993827,4cyclictest20055-21apt-get21:10:130
1693993718,11cyclictest35966irq/58-eth022:55:143
168599371,7cyclictest22322-21cut21:10:222
168599371,3cyclictest10713-21ssh23:10:122
166699378,9cyclictest31499-21df_inode22:10:140
1666993728,4cyclictest1819-21apt-get19:10:120
166699371,28cyclictest13230-21ssh22:00:140
166699371,28cyclictest13230-21ssh22:00:140
168599369,24cyclictest4716-21irqrtprio20:20:172
168599364,11cyclictest27346irq/40-dwc2_hso22:40:142
168599361,8cyclictest29093-21cat23:55:202
168599361,7cyclictest9673-21ps20:25:172
168599361,31cyclictest1010-21ps21:35:192
168599361,2cyclictest17452-21apt-get20:00:142
168599360,8cyclictest25220-21latency_hist00:30:012
167699362,23cyclictest22273-21ssh22:05:131
169399359,20cyclictest35966irq/58-eth019:40:123
169399354,14cyclictest35966irq/58-eth023:15:133
169399353,19cyclictest35966irq/58-eth000:25:113
169399351,23cyclictest35966irq/58-eth021:35:143
168599353,7cyclictest27346irq/40-dwc2_hso23:20:122
168599351,8cyclictest7587-21latency_hist19:50:012
168599351,8cyclictest5605-21ssh23:25:142
168599351,8cyclictest349-21diskstats23:40:172
168599351,8cyclictest28163-21cut23:00:232
168599351,8cyclictest28163-21cut23:00:232
168599351,7cyclictest24724-21timerandwakeup19:30:272
168599351,7cyclictest18588-21proc_pri21:05:242
168599350,8cyclictest1743-21basename00:35:012
167699352,24cyclictest6264-21latency19:45:211
1666993521,11cyclictest32681-21apt-get23:40:150
169399343,21cyclictest35966irq/58-eth023:05:123
168599347,3cyclictest0-21swapper/221:15:282
168599341,8cyclictest17341-21wc20:30:262
168599341,7cyclictest16309-21diskstats19:25:132
168599341,6cyclictest22008-21sort00:10:122
168599341,6cyclictest18610-21ls23:30:292
168599341,4cyclictest16322-21ssh00:25:022
168599341,3cyclictest0-21swapper/223:36:162
168599341,10cyclictest13423-21sort22:00:172
168599341,10cyclictest13423-21sort22:00:172
167699343,24cyclictest25583-21ssh00:30:131
166699342,24cyclictest28883-21cut19:35:210
168599339,22cyclictest8038-21ssh22:30:542
168599333,4cyclictest0-21swapper/223:45:132
1685993328,3cyclictest27-21ksoftirqd/221:25:132
168599331,7cyclictest28065-21latency_hist20:45:012
168599331,6cyclictest22308-21timerandwakeup20:35:272
168599331,10cyclictest13261-21sed00:05:172
168599331,10cyclictest12580-21/usr/sbin/munin19:55:162
168599330,7cyclictest31796-21apt-get20:15:122
168599330,6cyclictest31771-21cat21:50:292
167699333,19cyclictest9301-21ssh23:45:131
167699332,23cyclictest15948-21ls19:55:211
166699339,10cyclictest25521-21ssh21:48:090
166699336,7cyclictest31594-21ntp_192.168.11420:45:230
1666993322,8cyclictest17094-21apt-get22:20:130
168599325,4cyclictest8384-21irqrtprio19:50:172
168599323,6cyclictest10091-21cat21:20:312
168599321,6cyclictest10723-21munin-run00:40:012
168599321,5cyclictest468-21cat20:50:022
168599320,8cyclictest10548-21ntp_kernel_err22:50:272
167699322,22cyclictest23402-21/usr/sbin/munin20:40:211
167699322,21cyclictest28884-21tr19:35:211
167699321,18cyclictest6881-21ssh22:50:141
166699328,8cyclictest21115-21df_inode19:30:140
166699325,9cyclictest337-21ssh22:27:350
166699325,9cyclictest337-21ssh22:27:350
1693993116,10cyclictest35966irq/58-eth023:45:133
168599314,5cyclictest10556-21sh21:57:202
168599310,6cyclictest8044-21ssh23:07:152
168599310,4cyclictest31364-21apt-get22:10:132
167699312,21cyclictest9205-21kthreadcore19:15:211
166699317,14cyclictest0-21swapper/000:37:140
1666993111,16cyclictest0-21swapper/021:15:150
169399305,18cyclictest35966irq/58-eth020:05:133
168599301,6cyclictest0-21swapper/222:55:142
167699304,17cyclictest21-21ksoftirqd/121:40:141
166699307,9cyclictest19661-21diskmemload21:59:340
166699306,8cyclictest30987-21munin-run22:10:000
169399296,11cyclictest35966irq/58-eth022:30:143
168599291,10cyclictest27009-21sort20:10:112
167699292,20cyclictest30207-21ssh22:45:141
167699292,19cyclictest2333-21kthreadcore20:15:211
167699292,18cyclictest15957-21ssh22:55:121
167699292,17cyclictest29128-21ssh21:15:151
167699291,21cyclictest5734-21ssh21:20:121
166699299,6cyclictest19661-21diskmemload00:21:030
166699295,8cyclictest2795-21ntp_192.168.11420:15:230
166699292,21cyclictest21146-21ssh22:40:140
166699292,19cyclictest9206-21kthreadcore19:15:200
1666992921,5cyclictest2844-21apt-get22:30:130
1666992910,8cyclictest12868-21uname23:45:230
169399283,17cyclictest0-21swapper/319:55:213
169399283,16cyclictest0-21swapper/321:30:213
1693992811,11cyclictest0-21swapper/322:03:213
1693992811,11cyclictest0-21swapper/322:03:203
167699283,19cyclictest1688-21ssh23:05:131
167699282,20cyclictest28927-21grep22:25:261
167699282,20cyclictest28927-21grep22:25:261
166699288,7cyclictest19661-21diskmemload22:55:000
166699286,8cyclictest19661-21diskmemload00:10:340
166699285,8cyclictest25937-21ssh00:30:170
166699285,8cyclictest11820-21ssh23:27:100
166699283,9cyclictest0-21swapper/023:19:500
166699283,20cyclictest31032-21ssh00:15:120
166699282,21cyclictest13002-21ssh00:05:140
166699282,19cyclictest6263-21latency19:45:210
1666992818,7cyclictest20194-21ntp_192.168.11400:25:240
1666992812,12cyclictest4444-21kthreadcore23:05:210
169399273,18cyclictest0-21swapper/322:25:263
169399273,18cyclictest0-21swapper/322:25:253
169399273,16cyclictest0-21swapper/321:57:293
167699274,17cyclictest32695-21ssh23:40:151
166699275,8cyclictest11033-21sh21:21:000
166699275,15cyclictest31600-21sh21:34:020
166699274,7cyclictest30179-21apt-get22:45:130
1666992722,3cyclictest12006-21apt-get22:35:140
166699272,19cyclictest12824-21grep21:40:220
1666992714,10cyclictest27469-21apt-get21:50:120
1666992713,10cyclictest7688-21ntp_states00:00:270
1666992713,10cyclictest0-21swapper/020:20:100
166699271,17cyclictest14581-21ssh23:30:120
169399263,16cyclictest0-21swapper/322:20:213
169399263,15cyclictest0-21swapper/319:15:213
1693992610,11cyclictest0-21swapper/322:43:253
167699263,8cyclictest0-21swapper/123:53:261
167699262,19cyclictest3998-21ssh00:00:141
167699261,13cyclictest13212-21ssh22:00:131
167699261,13cyclictest13212-21ssh22:00:131
1693992512,8cyclictest35966irq/58-eth000:20:133
169399251,15cyclictest35966irq/58-eth022:15:123
1676992510,9cyclictest1657-21cyclictest21:35:141
166699256,8cyclictest18685-21/usr/sbin/munin20:35:140
166699254,8cyclictest372-21apt-get21:35:120
1666992516,6cyclictest11433-21apt-get19:20:130
1666992516,4cyclictest22234-21apt-get20:05:130
1666992515,7cyclictest30467-21ntp_states20:10:230
1666992514,7cyclictest16095-21ntp_states19:55:240
1666992513,8cyclictest30728-21diskstats19:40:150
1666992512,7cyclictest0-21swapper/023:50:120
1666992511,10cyclictest7907-21diskstats19:50:140
169399243,14cyclictest0-21swapper/320:45:223
167699242,17cyclictest12005-21ssh22:35:131
1676992410,5cyclictest0-21swapper/122:10:121
166699246,12cyclictest0-21swapper/019:25:120
1666992418,3cyclictest15103-21apt-get21:05:110
1666992412,9cyclictest20774-21sed20:00:210
1666992412,9cyclictest15542-21kthreadcore20:30:200
1666992411,10cyclictest2510-21cut20:50:200
169399232,14cyclictest35966irq/58-eth000:05:133
169399231,15cyclictest0-21swapper/321:25:203
1693992310,8cyclictest35966irq/58-eth022:45:133
167699237,4cyclictest131rcu_preempt00:00:021
167699232,16cyclictest19661-21diskmemload21:53:201
167699230,18cyclictest21-21ksoftirqd/120:25:111
169399223,13cyclictest33-21ksoftirqd/323:10:113
169399223,13cyclictest0-21swapper/321:50:213
167699229,8cyclictest2867-21sh22:30:141
167699223,14cyclictest7625-21ssh00:20:131
167699221,17cyclictest16628-21ssh00:05:261
167699221,16cyclictest8067-21ssh22:15:141
167699221,15cyclictest5615-21ls20:55:161
1666992213,6cyclictest8791-21ntp_192.168.11420:55:230
169399219,7cyclictest35966irq/58-eth021:45:233
169399218,7cyclictest321rcuc/323:50:123
1693992110,5cyclictest0-21swapper/300:10:213
1676992112,6cyclictest0-21swapper/120:45:141
167699211,17cyclictest7522-21ls20:20:221
167699211,17cyclictest21282-21cut22:40:171
167699211,16cyclictest16691-21ls00:25:131
167699211,15cyclictest10391-21uniq21:00:151
1666992111,7cyclictest23504-21df20:40:140
1666992110,9cyclictest1657-21cyclictest19:10:010
169399209,8cyclictest0-21swapper/322:05:113
169399209,7cyclictest35966irq/58-eth022:10:133
169399202,12cyclictest35966irq/58-eth023:30:113
169399202,12cyclictest35966irq/58-eth023:20:133
1693992011,6cyclictest35966irq/58-eth023:35:133
169399201,14cyclictest0-21swapper/321:05:013
1693992010,8cyclictest0-21swapper/300:33:383
167699209,8cyclictest30894-21cut23:20:211
167699202,9cyclictest14066-21kthreadcore23:10:221
1676992013,5cyclictest31568-21munin-plugin-st20:15:011
1676992011,6cyclictest25403-21sh23:00:181
1676992011,6cyclictest25403-21sh23:00:171
1676992011,6cyclictest0-21swapper/121:25:131
167699200,16cyclictest11542-21cut19:20:141
169399199,8cyclictest0-21swapper/321:22:213
169399193,9cyclictest0-21swapper/319:45:213
169399193,11cyclictest0-21swapper/320:50:213
169399193,11cyclictest0-21swapper/320:20:213
169399193,11cyclictest0-21swapper/320:10:213
169399193,11cyclictest0-21swapper/320:00:213
169399193,11cyclictest0-21swapper/319:30:213
169399193,11cyclictest0-21swapper/319:10:213
169399192,6cyclictest0-21swapper/320:40:213
169399192,11cyclictest0-21swapper/321:05:043
1693991913,4cyclictest0-21swapper/323:40:173
1693991910,4cyclictest0-21swapper/300:17:563
167699191,7cyclictest22567-21irqcore00:10:181
167699191,15cyclictest5556-21ssh23:25:131
169399183,9cyclictest0-21swapper/320:15:213
169399183,9cyclictest0-21swapper/319:35:213
169399183,10cyclictest0-21swapper/319:25:213
169399182,10cyclictest35966irq/58-eth022:50:153
169399182,10cyclictest0-21swapper/320:35:133
1693991811,5cyclictest0-21swapper/321:15:133
1693991811,4cyclictest0-21swapper/300:02:413
169399181,12cyclictest0-21swapper/300:35:263
1693991810,6cyclictest0-21swapper/323:00:223
1693991810,6cyclictest0-21swapper/323:00:223
1693991810,4cyclictest0-21swapper/320:30:133
167699189,6cyclictest614-21dbus-daemon19:11:591
167699189,6cyclictest24837-21grep19:30:271
167699187,6cyclictest21111-21timerandwakeup20:00:281
167699183,10cyclictest0-21swapper/120:50:131
167699181,8cyclictest16471-21kthreadcore20:30:201
167699181,14cyclictest16069-21kthreadcore21:05:171
1693991711,4cyclictest0-21swapper/322:35:183
169399171,10cyclictest0-21swapper/321:10:143
167699179,5cyclictest19687-21kthreadcore20:35:171
167699175,10cyclictest22341-21grep20:05:161
167699174,4cyclictest31519-21kthreadcore19:40:191
167699171,8cyclictest14549-21/usr/sbin/munin23:30:111
169399169,5cyclictest0-21swapper/319:22:413
168599160,8cyclictest1686-21latency_hist19:10:012
167699169,5cyclictest18789-21kthreadcore19:25:191
167699161,10cyclictest0-21swapper/119:50:131
169399141,9cyclictest0-21swapper/320:25:123
167699141,9cyclictest1766-21latency_hist19:10:011
169399132,6cyclictest0-21swapper/319:50:203
169399131,5cyclictest0-21swapper/320:55:133
169399101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional