You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-03 - 13:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Fri Apr 03, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
511599682,53cyclictest31337-21ssh21:50:140
511599621,55cyclictest19690-21rm22:55:130
513999615,32cyclictest30043-21ssh22:25:142
5147995733,15cyclictest28366irq/58-eth000:30:163
512999565,43cyclictest27451-21ssh21:30:131
512999565,43cyclictest27451-21ssh21:30:131
5147995030,12cyclictest28366irq/58-eth022:45:143
5139995027,14cyclictest0-21swapper/200:05:132
513999482,25cyclictest2130-21sh00:15:132
511599481,38cyclictest17354-21apt-get20:30:110
5147994727,12cyclictest28366irq/58-eth021:20:133
511599472,38cyclictest14904-21apt-get19:20:110
512999464,33cyclictest2880-21ssh22:10:121
512999462,5cyclictest17121-21ssh22:00:121
511599461,38cyclictest20149-21apt-get00:25:130
512999451,22cyclictest29116-21ssh00:30:161
511599453,32cyclictest18233-21ssh23:30:110
511599452,37cyclictest20952-21apt-get20:00:110
511599452,35cyclictest16602-21apt-get00:05:130
511599451,38cyclictest13703-21apt-get21:00:110
511599442,37cyclictest0-21swapper/022:20:140
511599441,40cyclictest4064-21ssh21:35:120
511599441,37cyclictest2911-21apt-get20:15:140
513999439,16cyclictest27-21ksoftirqd/219:35:122
511599431,34cyclictest24492-21apt-get19:30:110
511599421,37cyclictest22135-21apt-get20:35:120
511599421,37cyclictest0-21swapper/021:55:110
511599421,36cyclictest5107-21cyclictest20:10:120
511599421,35cyclictest29294-21apt-get19:35:120
511599421,2cyclictest3849-21apt-get23:40:130
511599421,2cyclictest0-21swapper/021:40:130
511599421,25cyclictest0-21swapper/020:55:100
514799412,25cyclictest28366irq/58-eth000:20:133
5139994121,16cyclictest27-21ksoftirqd/200:00:122
511599412,35cyclictest2125-21apt-get00:15:140
511599412,34cyclictest1668-21apt-get19:40:110
512999402,25cyclictest11915-21ssh22:15:141
512999402,12cyclictest25550-21ssh00:10:141
511599401,35cyclictest32581-21apt-get21:15:150
511599401,2cyclictest0-21swapper/022:40:130
511599400,2cyclictest7623-21ssh00:00:130
511599400,27cyclictest8725-21ssh23:24:580
5147993919,11cyclictest0-21swapper/322:50:113
511599391,26cyclictest1813-21rm23:38:030
511599383,29cyclictest26963-21apt-get20:40:120
513999372,28cyclictest23250-21rm23:30:432
5139993722,7cyclictest0-21swapper/219:50:132
513999372,19cyclictest1718-21ssh23:37:562
511599377,26cyclictest13869-21cat23:25:300
511599373,30cyclictest0-21swapper/022:50:090
511599372,23cyclictest5107-21cyclictest22:26:230
511599371,33cyclictest0-21swapper/020:05:120
511599371,2cyclictest0-21swapper/021:10:140
511599371,24cyclictest12665-21sh22:31:500
511599371,24cyclictest12665-21sh22:31:500
513999364,18cyclictest22346irq/40-dwc2_hso20:40:122
512999367,6cyclictest27110-21cut22:05:191
512999361,12cyclictest18047-21ntp_kernel_pll_23:10:251
511599362,32cyclictest0-21swapper/022:10:140
511599362,30cyclictest5351-21apt-get23:05:130
511599361,30cyclictest10102-21apt-get19:15:110
511599361,29cyclictest15767-21apt-get22:35:140
511599360,33cyclictest1550-21ssh22:45:130
513999352,26cyclictest3875-21ssh23:40:142
5139993518,11cyclictest0-21swapper/223:55:112
512999351,25cyclictest20984-21ssh22:20:131
512999351,10cyclictest23772-21ntp_192.168.11400:25:251
511599351,31cyclictest0-21swapper/000:35:120
511599351,30cyclictest0-21swapper/000:10:130
511599351,2cyclictest17102-21apt-get22:00:120
511599351,2cyclictest0-21swapper/022:05:130
511599351,29cyclictest31727-21apt-get20:45:110
511599351,28cyclictest26970-21apt-get21:30:020
511599351,28cyclictest26970-21apt-get21:30:020
511599350,2cyclictest21953-21apt-get23:50:120
511599350,19cyclictest22644-21ssh23:14:340
512999341,15cyclictest4616-21irqcore21:35:171
512999341,10cyclictest4275-21kthreadcore00:15:201
511599342,27cyclictest31016-21apt-get23:55:120
511599341,30cyclictest11165-21ssh00:20:120
511599341,29cyclictest9881-21memory19:45:210
511599341,29cyclictest12691-21uniq20:25:170
511599340,4cyclictest19711-21apt-get19:25:130
511599340,30cyclictest13869-21kthreadcore22:15:210
513999332,21cyclictest22346irq/40-dwc2_hso20:35:122
513999332,18cyclictest19689-21sh22:55:142
512999332,10cyclictest22125-21kthreadcore22:55:211
511599331,3cyclictest9183-21apt-get21:20:130
511599331,29cyclictest5339-21rm00:35:010
511599331,29cyclictest29713-21ssh23:17:110
511599331,28cyclictest7684-21swap20:50:250
511599330,29cyclictest19870-21timerwakeupswit19:55:270
5147993212,11cyclictest28366irq/58-eth023:15:123
512999329,3cyclictest5563-21cut23:05:151
512999329,3cyclictest23587-21kthreadcore20:00:191
512999322,24cyclictest8516-21ssh23:24:421
512999321,8cyclictest18278-21ssh21:25:131
5129993211,6cyclictest28028-21sendmail_mailqu19:30:251
512999321,10cyclictest30478-21kthreadcore23:00:201
511599321,28cyclictest18251-21apt-get21:25:120
511599321,28cyclictest15915-21sshd19:54:400
511599321,27cyclictest12952-21apt-get23:45:140
511599320,29cyclictest8413-21irqstats20:20:170
5147993117,8cyclictest28366irq/58-eth023:30:103
513999315,16cyclictest22346irq/40-dwc2_hso19:20:112
513999314,23cyclictest0-21swapper/223:05:132
512999313,11cyclictest23450-21ssh23:15:151
512999311,9cyclictest9160-21/usr/sbin/munin21:20:231
512999311,11cyclictest17860-21idleruntime-cro23:30:021
512999311,11cyclictest16495-21ssh22:35:201
512999311,10cyclictest10073-21grep19:45:261
511599311,27cyclictest28826-21awk23:00:130
511599311,27cyclictest22195-21apt-get21:45:110
511599311,26cyclictest23114-21apt-key21:10:010
512999302,5cyclictest0-21swapper/123:42:571
512999301,9cyclictest10437-21latency_hist22:50:021
512999301,12cyclictest11453-21mailstats00:00:281
512999301,10cyclictest7962-21ssh21:55:121
512999301,10cyclictest23081-21diskmemload21:10:241
512999301,10cyclictest16248-21diskstats19:55:161
511599301,26cyclictest5375-21df19:10:140
514799293,19cyclictest0-21swapper/323:40:423
514799293,19cyclictest0-21swapper/300:12:573
513999294,10cyclictest32452-21ssh00:12:572
513999293,19cyclictest22346irq/40-dwc2_hso20:25:152
513999292,21cyclictest22751-21ssh23:14:402
513999292,11cyclictest23430-21ssh23:15:122
512999298,5cyclictest28360-21latency22:40:231
514799285,10cyclictest0-21swapper/322:35:143
514799283,14cyclictest0-21swapper/321:30:133
514799283,14cyclictest0-21swapper/321:30:123
5147992811,13cyclictest0-21swapper/321:50:143
513999283,19cyclictest9210-21ssh21:20:132
512999287,5cyclictest23081-21diskmemload00:05:171
512999286,6cyclictest12098-21kthreadcore19:50:191
512999281,10cyclictest26148-21cat21:45:281
514799279,14cyclictest0-21swapper/322:20:143
514799273,20cyclictest28366irq/58-eth021:35:113
514799273,17cyclictest0-21swapper/323:23:423
514799272,12cyclictest28366irq/58-eth020:10:123
5147992715,8cyclictest0-21swapper/321:55:113
514799271,18cyclictest0-21swapper/323:37:563
514799271,13cyclictest28366irq/58-eth021:10:143
5147992711,10cyclictest28366irq/58-eth000:25:143
5147992711,10cyclictest0-21swapper/323:26:423
513999277,12cyclictest0-21swapper/200:35:132
513999275,13cyclictest22346irq/40-dwc2_hso20:10:122
513999272,16cyclictest22346irq/40-dwc2_hso19:10:112
513999272,15cyclictest0-21swapper/221:00:112
5139992713,4cyclictest14243-21ssh22:50:212
512999271,12cyclictest7766-21kthreadcore22:30:201
512999271,12cyclictest7766-21kthreadcore22:30:201
511599272,23cyclictest5107-21cyclictest19:10:010
513999262,19cyclictest0-21swapper/223:26:422
513999262,18cyclictest0-21swapper/219:30:122
5139992613,8cyclictest27-21ksoftirqd/223:20:422
512999261,9cyclictest1764-21/usr/sbin/munin22:25:311
512999261,9cyclictest13181-21/usr/sbin/munin21:40:281
514799253,15cyclictest0-21swapper/300:15:433
514799252,13cyclictest0-21swapper/323:55:133
5147992514,5cyclictest28366irq/58-eth022:40:153
513999254,14cyclictest22346irq/40-dwc2_hso22:45:142
513999253,15cyclictest22346irq/40-dwc2_hso20:15:122
513999253,13cyclictest22346irq/40-dwc2_hso00:25:142
513999252,10cyclictest32600-21ssh21:15:162
513999251,21cyclictest13502-21ls21:40:172
514799248,10cyclictest0-21swapper/321:15:153
5147992412,6cyclictest28366irq/58-eth000:05:143
513999243,12cyclictest22346irq/40-dwc2_hso20:50:102
513999242,17cyclictest22047-21users22:00:312
513999242,17cyclictest12979-21ssh23:45:142
513999241,14cyclictest24893-21sh22:40:142
512999242,8cyclictest5870-21diskstats00:35:151
514799236,13cyclictest0-21swapper/322:00:123
514799233,9cyclictest0-21swapper/320:55:123
513999238,6cyclictest21581-21irqrtprio22:20:202
513999236,14cyclictest11500-21wc20:20:272
513999232,6cyclictest0-21swapper/200:23:002
513999232,4cyclictest0-21swapper/219:45:112
513999232,18cyclictest26387-21fschecks_count22:05:152
513999232,17cyclictest469-21grep00:30:312
513999231,4cyclictest0-21swapper/220:00:132
514799229,9cyclictest28366irq/58-eth022:55:133
514799223,13cyclictest0-21swapper/323:10:413
514799221,13cyclictest28366irq/58-eth021:40:133
513999227,12cyclictest0-21swapper/221:40:022
513999224,11cyclictest27-21ksoftirqd/221:10:142
513999223,11cyclictest22972-21grep21:45:182
513999221,15cyclictest16879-21ls22:00:022
513999221,15cyclictest0-21swapper/221:25:132
512999224,5cyclictest131rcu_preempt20:45:111
512999221,14cyclictest23080-21latency_hist21:10:021
514799219,9cyclictest0-21swapper/323:00:073
5147992116,3cyclictest321rcuc/322:07:513
5147992113,5cyclictest28366irq/58-eth022:10:133
514799211,13cyclictest0-21swapper/319:45:013
5147992110,8cyclictest0-21swapper/300:04:153
513999216,13cyclictest1456-21kthreadcore20:45:202
513999214,9cyclictest3170-21/usr/sbin/munin22:10:172
513999212,9cyclictest6917-21cut22:30:182
513999212,9cyclictest6917-21cut22:30:172
513999212,14cyclictest22346irq/40-dwc2_hso23:50:122
513999212,14cyclictest22346irq/40-dwc2_hso19:15:122
513999211,12cyclictest9671-21kthreadcore20:55:182
513999211,10cyclictest20979-21switchtime20:30:262
5139992110,9cyclictest3612-21ssh23:03:242
513999210,12cyclictest21986-21kthreadcore19:25:202
5129992114,4cyclictest0-21swapper/100:20:121
514799209,8cyclictest0-21swapper/322:28:433
514799205,12cyclictest0-21swapper/323:45:143
513999207,10cyclictest23081-21diskmemload22:35:212
513999203,14cyclictest28998-21wc20:05:222
513999203,14cyclictest21927-21awk21:05:272
513999203,10cyclictest26970-21apt-get21:30:042
513999203,10cyclictest26970-21apt-get21:30:042
513999201,11cyclictest14442-21kthreadcore22:15:222
513999201,10cyclictest3378-21kthreadcore19:40:212
513999201,10cyclictest3083-21usage_Anybus21:50:312
5129992013,4cyclictest0-21swapper/120:30:261
512999201,12cyclictest31340-21sh21:50:151
512999200,17cyclictest31179-21cut23:55:141
514799195,11cyclictest0-21swapper/300:35:133
514799192,9cyclictest0-21swapper/320:20:113
514799192,10cyclictest0-21swapper/321:00:133
5147991914,2cyclictest0-21swapper/319:35:123
5147991913,4cyclictest0-21swapper/319:50:263
5147991911,6cyclictest0-21swapper/323:53:123
5147991911,6cyclictest0-21swapper/321:25:233
513999196,5cyclictest18352-21kthreadcore19:55:202
512999191,8cyclictest14971-21cat22:50:271
512999191,6cyclictest30577-21munin-run23:55:011
5129991914,3cyclictest27225-21/usr/sbin/munin23:35:271
512999191,15cyclictest30224-21latency20:40:211
512999190,15cyclictest10093-21find19:15:111
514799189,6cyclictest28366irq/58-eth021:45:113
514799188,8cyclictest0-21swapper/322:18:513
514799183,10cyclictest0-21swapper/323:07:533
514799182,10cyclictest0-21swapper/321:05:153
514799182,10cyclictest0-21swapper/320:30:123
514799182,10cyclictest0-21swapper/320:25:143
514799182,10cyclictest0-21swapper/319:25:133
5147991812,4cyclictest0-21swapper/322:30:163
5147991812,4cyclictest0-21swapper/322:30:153
512999183,10cyclictest18213-21ssh23:30:101
512999182,11cyclictest750-21sendmail-mta20:20:121
5129991812,4cyclictest0-21swapper/121:15:181
5129991811,5cyclictest0-21swapper/120:35:211
512999181,13cyclictest12978-21ssh23:45:141
514799179,5cyclictest28366irq/58-eth019:55:133
514799172,10cyclictest0-21swapper/320:45:133
514799172,10cyclictest0-21swapper/320:35:123
5147991712,3cyclictest0-21swapper/320:50:223
5147991712,3cyclictest0-21swapper/320:15:143
5147991712,3cyclictest0-21swapper/319:30:253
514799171,10cyclictest0-21swapper/319:15:133
512999179,6cyclictest13927-21wc21:00:141
512999179,6cyclictest1-21systemd20:52:381
512999179,5cyclictest20589-21kthreadcore19:25:181
512999178,6cyclictest2670-21sshd20:13:401
512999177,8cyclictest16013-21ntp_states20:25:261
512999176,8cyclictest1-21systemd20:57:381
512999174,10cyclictest6263-21munin-run19:45:011
5129991710,5cyclictest18391-21open_inodes19:20:241
514799168,6cyclictest0-21swapper/319:22:103
514799162,10cyclictest0-21swapper/319:45:133
5147991611,3cyclictest0-21swapper/320:00:123
5147991610,4cyclictest321rcuc/320:40:143
512999168,6cyclictest29388-21diskstats19:35:151
512999161,11cyclictest3344-21irqcore20:15:171
512999161,11cyclictest30457-21cat20:10:021
5129991610,4cyclictest0-21swapper/119:10:271
5147991510,3cyclictest0-21swapper/320:05:263
513999151,10cyclictest5237-21latency_hist19:10:022
512999153,9cyclictest5160-21latency_hist19:10:011
514799149,3cyclictest0-21swapper/319:10:503
514799111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional