You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-20 - 06:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Mon Apr 20, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2056699576,45cyclictest19181-21ssh21:35:140
20566995546,4cyclictest1263-21sh21:25:150
2056699543,12cyclictest131rcu_preempt21:15:140
2059399484,25cyclictest35966irq/58-eth022:45:123
2057699481,3cyclictest0-21swapper/121:05:131
20566994823,19cyclictest12-21ksoftirqd/022:15:120
2057699475,30cyclictest32166-21ssh00:25:161
2057699462,5cyclictest0-21swapper/119:30:131
2057699461,7cyclictest29362-21ssh22:35:131
2059399444,26cyclictest35966irq/58-eth000:25:153
2059399442,25cyclictest35966irq/58-eth023:00:153
2058599443,34cyclictest1672-21ssh23:50:122
2057699431,6cyclictest30646-21apt-get23:30:111
2057699431,6cyclictest0-21swapper/120:00:131
2057699421,2cyclictest0-21swapper/121:50:131
2057699421,2cyclictest0-21swapper/121:50:131
2057699401,6cyclictest0-21swapper/121:00:151
2057699401,6cyclictest0-21swapper/121:00:141
2058599392,27cyclictest930-21snmpd19:55:142
2057699392,5cyclictest0-21swapper/120:40:121
2057699391,5cyclictest0-21swapper/120:25:141
2057699391,5cyclictest0-21swapper/120:25:141
2059399382,27cyclictest35966irq/58-eth022:05:133
20593993811,18cyclictest35966irq/58-eth000:10:133
20593993811,18cyclictest35966irq/58-eth000:10:133
2057699381,3cyclictest0-21swapper/120:05:131
2057699372,30cyclictest1690-21ssh23:50:131
2057699372,25cyclictest21704-21ssh23:25:141
2057699372,25cyclictest21704-21ssh23:25:141
2057699372,13cyclictest5885-21ssh22:40:151
2057699372,11cyclictest9355-21irqstats00:30:201
2056699376,24cyclictest25089-21ssh23:45:120
20585993612,12cyclictest27-21ksoftirqd/222:35:122
2057699362,7cyclictest0-21swapper/120:15:141
2057699362,4cyclictest0-21swapper/120:45:131
2057699362,3cyclictest20425-21apt-get22:30:131
20576993620,10cyclictest6148-21diskmemload21:30:151
2057699361,6cyclictest13475-21/usr/sbin/munin20:10:241
2057699361,5cyclictest0-21swapper/119:40:121
2056699366,26cyclictest5255-21ssh00:10:130
2056699366,26cyclictest5255-21ssh00:10:130
2056699362,26cyclictest32730-21ssh22:55:140
20566993612,18cyclictest12-21ksoftirqd/019:50:140
2059399353,19cyclictest35966irq/58-eth021:20:143
2059399353,19cyclictest35966irq/58-eth021:20:143
2057699351,9cyclictest6623-21grep22:20:311
2057699351,9cyclictest20646-21kthreadcore00:35:241
2057699351,10cyclictest11442-21irqstats23:55:191
2056699352,29cyclictest8656-21ssh00:30:150
20566993516,10cyclictest12-21ksoftirqd/023:20:140
2057699341,8cyclictest32679-21/usr/sbin/munin22:55:151
2056699342,19cyclictest8007-21ssh22:05:120
2059399331,21cyclictest35966irq/58-eth000:20:133
2058599335,20cyclictest930-21snmpd20:10:132
2057699338,22cyclictest6630-21tr20:35:211
2057699338,22cyclictest5406-21fschecks_count00:10:161
2057699338,22cyclictest5406-21fschecks_count00:10:161
2057699337,22cyclictest7641-21switchtime23:15:261
2057699332,8cyclictest29414-21kthreadcore23:10:211
2057699332,8cyclictest27755-21sendmail-msp21:40:021
2057699332,26cyclictest0-21swapper/121:43:541
2057699331,9cyclictest7104-21irqcore21:10:191
2057699331,9cyclictest32492-21/usr/sbin/munin00:05:281
2057699331,9cyclictest28294-21ssh21:20:261
2057699331,9cyclictest28294-21ssh21:20:261
2057699331,9cyclictest2490-21latency_hist19:25:001
2057699331,8cyclictest6464-21timerwakeupswit19:25:291
2057699331,8cyclictest27498-21cut22:15:211
2057699331,8cyclictest27206-21kthreadcore19:15:201
2057699331,8cyclictest1683-21tr22:00:211
2057699331,8cyclictest1457-21latency_hist23:50:021
2057699331,7cyclictest20511-21tr23:05:221
2057699331,6cyclictest25789-21kthreadcore20:20:231
2057699331,5cyclictest1239-21apt-get21:25:151
2057699331,10cyclictest7151-21ssh23:35:131
2057699331,10cyclictest19745-21ntp_192.168.11423:40:241
2056699332,22cyclictest14845-21ssh22:45:140
20566993317,9cyclictest24657-21ssh21:20:140
20566993317,9cyclictest24657-21ssh21:20:130
2059399322,19cyclictest35966irq/58-eth022:10:143
2058599323,23cyclictest25095-21ssh23:45:132
2057699321,9cyclictest15743-21sort21:15:161
2057699321,9cyclictest11574-21tr23:00:221
2057699321,8cyclictest31096-21sendmail22:00:011
2057699321,8cyclictest28868-21apt21:00:011
2057699321,8cyclictest26285-21ssh00:02:331
2057699321,7cyclictest15483-21kthreadcore00:15:201
2057699321,6cyclictest18137-21kthreadcore22:10:201
2057699321,6cyclictest15280-21cut22:45:181
2057699321,6cyclictest14805-21kthreadcore19:35:211
2057699320,9cyclictest17127-21ssh22:26:281
2057699320,8cyclictest24252-21latency_hist20:55:001
2057699320,10cyclictest12692-21apt-get23:20:131
2059399312,19cyclictest35966irq/58-eth020:15:143
20585993121,8cyclictest27-21ksoftirqd/221:25:152
20585993116,11cyclictest20553-21cyclictest22:05:132
2057699312,4cyclictest25374-21cut00:20:221
2057699311,8cyclictest4562-21cat20:30:311
2057699311,8cyclictest12621-21cat22:05:291
2057699311,7cyclictest20783-21sort19:10:121
2057699311,5cyclictest23717-21/usr/sbin/munin22:50:311
2056699312,19cyclictest31515-21sh22:00:140
20566993117,10cyclictest0-21swapper/023:15:110
20566993114,11cyclictest0-21swapper/023:10:130
2059399304,16cyclictest35966irq/58-eth020:35:143
20593993018,8cyclictest35966irq/58-eth000:15:143
2057699301,8cyclictest26933-21sed19:50:171
2057699301,8cyclictest2653-21mailstats19:55:281
2057699301,7cyclictest7439-21cut21:45:221
2057699301,7cyclictest22552-21cut19:45:171
2056699300,22cyclictest0-21swapper/020:20:150
2058599293,16cyclictest14123-21ssh00:15:122
20585992925,2cyclictest27346irq/40-dwc2_hso23:00:142
2058599292,16cyclictest27346irq/40-dwc2_hso20:05:132
20585992910,16cyclictest0-21swapper/222:45:142
2057699290,6cyclictest20621-21sed19:10:011
2056699292,4cyclictest0-21swapper/020:35:130
2056699292,23cyclictest10737-21ssh23:55:140
2059399283,18cyclictest0-21swapper/323:41:533
20593992814,10cyclictest35966irq/58-eth022:55:143
2058599281,19cyclictest24329-21rm22:30:302
2059399273,14cyclictest0-21swapper/321:25:153
2058599273,17cyclictest0-21swapper/223:25:142
2058599273,17cyclictest0-21swapper/223:25:142
2058599272,17cyclictest27346irq/40-dwc2_hso23:40:142
2058599272,17cyclictest25951-21ssh22:15:142
2058599272,16cyclictest930-21snmpd20:45:142
2058599271,16cyclictest0-21swapper/221:35:142
2058599271,11cyclictest93066snmpd20:35:172
2059399263,17cyclictest0-21swapper/321:45:143
2059399263,11cyclictest35966irq/58-eth022:20:153
2059399263,11cyclictest0-21swapper/300:38:533
2059399262,14cyclictest0-21swapper/323:25:143
2059399262,14cyclictest0-21swapper/323:25:133
2058599261,15cyclictest22528-21ssh21:55:152
20585992611,12cyclictest20039-21cron22:30:022
2056699263,12cyclictest23197-21ssh00:20:140
2059399254,17cyclictest35966irq/58-eth000:00:153
2059399253,17cyclictest0-21swapper/323:45:123
2059399252,11cyclictest35966irq/58-eth019:50:133
2058599252,13cyclictest27346irq/40-dwc2_hso19:40:132
2056699255,14cyclictest630-21Async09:36:310
2056699252,19cyclictest19770-21ssh00:00:160
2056699251,17cyclictest4740-21ssh21:45:140
2059399242,15cyclictest35966irq/58-eth022:35:133
20593992410,11cyclictest0-21swapper/322:40:143
2058599246,3cyclictest131rcu_preempt21:45:282
2058599245,12cyclictest27346irq/40-dwc2_hso20:30:152
2058599242,8cyclictest0-21swapper/200:38:532
2058599242,16cyclictest0-21swapper/221:50:142
2058599242,16cyclictest0-21swapper/221:50:142
20585992413,8cyclictest0-21swapper/221:05:182
2058599241,12cyclictest261rcuc/221:15:142
2058599240,13cyclictest2531-21ls19:25:022
2056699242,18cyclictest1821-21ssh23:30:240
2056699242,15cyclictest17686-21ssh00:35:160
2056699241,16cyclictest18246-21ssh23:05:140
2059399238,9cyclictest35966irq/58-eth021:55:143
2059399232,14cyclictest35966irq/58-eth019:45:113
2059399232,13cyclictest0-21swapper/323:17:163
2059399231,14cyclictest35966irq/58-eth023:05:123
2058599239,5cyclictest0-21swapper/219:10:242
2058599236,13cyclictest27346irq/40-dwc2_hso20:50:132
2058599234,16cyclictest30671-21ssh23:30:132
2058599232,6cyclictest0-21swapper/223:07:532
2058599232,6cyclictest0-21swapper/222:59:532
2058599232,16cyclictest0-21swapper/223:18:532
20585992311,9cyclictest12603-21ssh23:36:112
20585992310,4cyclictest3383-21latency_hist23:15:022
20585992310,11cyclictest24774-21df21:20:162
20585992310,11cyclictest24774-21df21:20:162
20585992310,10cyclictest28870-21sh21:00:012
2058599230,16cyclictest31727-21latency_hist00:25:022
2056699232,17cyclictest6148-21diskmemload23:01:520
2056699231,16cyclictest20442-21ssh22:30:140
2059399223,13cyclictest35966irq/58-eth019:55:143
2059399223,13cyclictest0-21swapper/322:00:543
2058599225,11cyclictest27346irq/40-dwc2_hso21:10:142
2058599222,6cyclictest0-21swapper/222:04:532
2058599222,12cyclictest930-21snmpd20:25:152
2058599222,12cyclictest930-21snmpd20:25:152
20585992214,5cyclictest30551-21ssh22:52:532
2058599221,17cyclictest10219-21sh21:30:152
20585992210,9cyclictest5183-21ssh00:10:092
20585992210,9cyclictest5183-21ssh00:10:092
2056699228,4cyclictest131rcu_preempt20:50:010
2056699227,9cyclictest20553-21cyclictest22:10:130
2056699223,11cyclictest0-21swapper/020:50:140
20566992211,8cyclictest31083-21cron22:00:010
2056699220,19cyclictest32442-21ls21:00:240
2056699220,19cyclictest32442-21ls21:00:240
2059399212,11cyclictest35966irq/58-eth019:40:133
20593992111,4cyclictest0-21swapper/322:25:343
2058599214,9cyclictest930-21snmpd23:57:492
2058599213,14cyclictest27346irq/40-dwc2_hso23:20:142
2058599213,10cyclictest6426-21switchtime22:20:302
2058599212,16cyclictest10730-21latency19:30:232
2058599212,15cyclictest27346irq/40-dwc2_hso20:20:142
2058599211,17cyclictest9500-21ntp_192.168.11422:40:262
2058599211,17cyclictest14504-21grep20:45:012
2058599211,11cyclictest1727-21kthreadcore00:25:232
2056699217,8cyclictest12-21ksoftirqd/000:05:140
2056699217,11cyclictest27585-21cat22:50:280
2056699211,18cyclictest32082-21timerandwakeup21:40:280
20566992111,7cyclictest12-21ksoftirqd/020:15:010
2056699211,16cyclictest25758-21ls19:15:160
20566992110,7cyclictest0-21swapper/019:10:120
2059399209,8cyclictest0-21swapper/321:11:113
2059399203,14cyclictest0-21swapper/323:55:143
2059399202,16cyclictest0-21swapper/322:30:133
2058599203,9cyclictest13432-21kthreadcore19:35:192
2058599201,9cyclictest433-21timerandwakeup21:00:302
2058599201,9cyclictest433-21timerandwakeup21:00:302
2058599201,15cyclictest930-21snmpd19:55:022
2058599201,15cyclictest8626-21latency_hist20:05:012
2058599201,15cyclictest3190-21ps19:25:182
2058599201,15cyclictest25592-21timerandwakeup19:45:272
2058599201,10cyclictest32034-21kthreadcore00:05:232
2058599201,10cyclictest30238-21latency_hist19:20:022
2058599201,10cyclictest19975-21sort00:00:182
20585992010,8cyclictest11517-21kthreadcore00:30:232
2056699201,4cyclictest0-21swapper/023:35:190
2056699201,16cyclictest21529-21ls20:15:230
2056699201,12cyclictest32224-21ls00:25:170
2056699200,17cyclictest1723-21cut21:05:170
2059399199,7cyclictest0-21swapper/323:50:303
2059399193,11cyclictest0-21swapper/321:54:533
2059399193,11cyclictest0-21swapper/321:54:523
2059399193,11cyclictest0-21swapper/321:44:533
2059399192,12cyclictest0-21swapper/321:30:153
2059399192,11cyclictest0-21swapper/322:15:143
20593991911,5cyclictest35966irq/58-eth000:05:143
2059399191,13cyclictest0-21swapper/322:50:243
2059399191,11cyclictest0-21swapper/321:05:013
2059399191,11cyclictest0-21swapper/321:05:013
2058599191,9cyclictest28289-21fschecks_count21:40:162
2058599191,15cyclictest18384-21fschecks_count20:15:172
2058599191,10cyclictest19494-21kthreadcore22:10:222
2056699193,9cyclictest0-21swapper/023:41:530
2056699192,7cyclictest4895-21munin-run20:35:020
2056699192,14cyclictest2959-21grep19:55:310
2056699191,9cyclictest23840-21kthreadcore23:25:210
2056699191,9cyclictest23840-21kthreadcore23:25:210
2056699191,3cyclictest0-21swapper/022:35:050
2056699191,10cyclictest0-21swapper/023:52:320
2059399189,6cyclictest0-21swapper/323:35:373
2059399189,6cyclictest0-21swapper/321:36:173
2059399183,12cyclictest0-21swapper/323:10:233
2059399182,10cyclictest0-21swapper/320:25:153
2059399182,10cyclictest0-21swapper/320:25:153
2059399182,10cyclictest0-21swapper/320:05:143
2059399182,10cyclictest0-21swapper/319:10:133
20593991811,4cyclictest0-21swapper/323:30:253
2059399181,13cyclictest0-21swapper/320:00:153
2059399181,12cyclictest0-21swapper/323:20:293
2059399181,10cyclictest0-21swapper/321:05:143
20593991810,6cyclictest0-21swapper/300:32:433
2058599181,14cyclictest20598-21cat19:10:012
2056699182,12cyclictest11601-21diskstats22:25:160
20566991812,4cyclictest0-21swapper/019:45:170
2056699181,15cyclictest28078-21fschecks_time20:25:170
2056699181,15cyclictest28078-21fschecks_time20:25:160
2056699181,14cyclictest17311-21fschecks_time19:40:160
2056699181,14cyclictest12544-21grep20:05:290
2056699181,12cyclictest7371-21users22:20:330
2059399179,6cyclictest0-21swapper/321:17:023
2059399172,9cyclictest0-21swapper/319:25:153
2059399172,10cyclictest0-21swapper/320:55:153
2059399172,10cyclictest0-21swapper/320:50:143
2059399172,10cyclictest0-21swapper/319:30:133
2059399171,12cyclictest0-21swapper/319:15:283
2056699172,12cyclictest26906-21kthreadcore20:55:220
2056699171,8cyclictest18893-21cat00:15:320
2056699171,13cyclictest6742-21kthreadcore20:00:220
2056699171,13cyclictest16911-21latency_hist19:40:020
2059399169,4cyclictest0-21swapper/320:10:133
2059399162,9cyclictest0-21swapper/320:30:153
2059399162,9cyclictest0-21swapper/320:20:143
20593991612,2cyclictest33-21ksoftirqd/319:35:243
2059399161,13cyclictest0-21swapper/320:40:163
2056699163,11cyclictest8174-21kthreadcore19:30:200
2056699162,2cyclictest131rcu_preempt22:40:140
2056699162,11cyclictest12159-21kthreadcore20:40:210
2056699161,5cyclictest10193-21apt-get21:30:130
2056699161,12cyclictest13809-21ssh21:50:160
2056699161,12cyclictest13809-21ssh21:50:150
2056699161,11cyclictest6738-21ssh21:10:150
2056699161,11cyclictest2921-21hwlatdetect19:25:170
2056699141,10cyclictest20553-21cyclictest19:10:010
2059399137,4cyclictest36066irq/59-eth020:45:163
2059399131,10cyclictest0-21swapper/319:20:203
2059399121,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional