You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-07 - 03:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Sun Dec 07, 2025 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2600199551,42cyclictest16271-21ssh00:15:150
26025995122,16cyclictest29566irq/58-eth023:05:133
2600199515,6cyclictest131rcu_preempt22:30:130
2600999483,35cyclictest24374-21ssh00:20:151
2600999481,38cyclictest28883-21ssh21:20:141
2602599474,26cyclictest29566irq/58-eth021:40:143
2600199471,40cyclictest20645-21ssh21:15:140
2600199464,8cyclictest131rcu_preempt00:35:130
2600199455,9cyclictest131rcu_preempt21:25:130
2600999432,32cyclictest3063-21ssh22:05:141
2600199433,32cyclictest32560-21ssh00:05:140
2602599423,25cyclictest29566irq/58-eth022:10:163
2600199422,29cyclictest26732-21ssh22:20:150
26009994127,3cyclictest21-21ksoftirqd/100:25:151
2600999391,31cyclictest8201-21ssh23:50:131
2600999391,31cyclictest8201-21ssh23:50:131
2600999391,30cyclictest2176-21ssh22:25:151
2600999391,30cyclictest2176-21ssh22:25:151
2600199391,6cyclictest131rcu_preempt20:50:170
2601799383,14cyclictest31890-21ssh00:25:162
2602599372,22cyclictest29566irq/58-eth000:05:143
26009993711,18cyclictest21-21ksoftirqd/123:45:131
2602599361,23cyclictest29566irq/58-eth019:30:143
2601799362,14cyclictest9426-21ssh22:50:132
26017993516,12cyclictest0-21swapper/222:20:152
2600999352,24cyclictest9057-21ssh23:10:141
2600999351,27cyclictest8574-21ssh23:30:131
2601799341,21cyclictest0-21swapper/220:10:162
2600999344,23cyclictest12473-21ssh21:30:141
2600199343,8cyclictest131rcu_preempt19:30:140
26025993317,12cyclictest29566irq/58-eth021:30:163
26017993313,3cyclictest131rcu_preempt20:00:152
2600199332,24cyclictest932-21snmpd20:00:160
26001993318,11cyclictest12-21ksoftirqd/019:15:120
26001993315,11cyclictest131rcu_preempt21:50:150
2601799321,23cyclictest564-21ssh23:25:132
2600199324,19cyclictest20523-21ssh21:35:130
2600199322,9cyclictest131rcu_preempt20:35:130
2601799316,17cyclictest26346irq/40-dwc2_hso19:15:132
2601799315,14cyclictest26346irq/40-dwc2_hso00:10:152
26017993122,5cyclictest24504-21sh00:00:152
26001993116,10cyclictest30040-1kworker/u9:223:50:140
26001993116,10cyclictest30040-1kworker/u9:223:50:140
2602599303,18cyclictest29566irq/58-eth023:55:133
26017993012,12cyclictest629-21Async14:00:482
2602599294,13cyclictest29566irq/58-eth023:20:143
2601799293,15cyclictest26346irq/40-dwc2_hso19:45:142
2601799292,19cyclictest26346irq/40-dwc2_hso22:45:142
2601799292,11cyclictest17875-21ssh22:35:112
2600999292,22cyclictest3918-21ssh21:45:151
26025992814,9cyclictest29566irq/58-eth021:55:123
2601799285,15cyclictest26346irq/40-dwc2_hso20:30:152
2601799284,13cyclictest26346irq/40-dwc2_hso19:20:132
2601799282,19cyclictest0-21swapper/221:00:152
2600199281,16cyclictest25541-21ssh23:00:130
2602599273,16cyclictest29566irq/58-eth023:25:113
26025992715,9cyclictest29566irq/58-eth023:00:143
2601799276,12cyclictest26346irq/40-dwc2_hso20:20:132
2601799276,12cyclictest13280-21kthreadcore22:10:242
2601799274,14cyclictest26346irq/40-dwc2_hso19:30:152
2601799272,12cyclictest12507-21ssh21:10:172
2600999279,12cyclictest21-21ksoftirqd/120:00:141
2600999273,17cyclictest24488-21ssh00:00:131
2600199271,21cyclictest27633-21ssh22:00:170
2601799264,17cyclictest26346irq/40-dwc2_hso20:55:142
2601799262,9cyclictest0-21swapper/219:50:152
2601799262,9cyclictest0-21swapper/219:50:152
26017992613,10cyclictest0-21swapper/221:20:272
2600199269,11cyclictest8537-21ssh23:09:500
2600199268,14cyclictest12-21ksoftirqd/021:20:150
26001992611,7cyclictest12-21ksoftirqd/021:30:160
2602599252,20cyclictest29566irq/58-eth021:35:133
26025992510,8cyclictest29566irq/58-eth022:25:173
26025992510,8cyclictest29566irq/58-eth022:25:163
26025992510,7cyclictest29566irq/58-eth023:50:153
26025992510,7cyclictest29566irq/58-eth023:50:143
2601799258,7cyclictest0-21swapper/222:05:142
2601799256,15cyclictest26346irq/40-dwc2_hso23:35:152
26017992514,8cyclictest0-21swapper/223:20:162
2600999253,17cyclictest18876-21ssh22:15:161
2600199253,14cyclictest8979-21kthreadcore19:25:210
2600199253,14cyclictest8979-21kthreadcore19:25:200
2600199252,15cyclictest12-21ksoftirqd/022:50:120
2600199251,7cyclictest131rcu_preempt20:25:130
2601799245,8cyclictest27-21ksoftirqd/223:52:242
2601799245,8cyclictest27-21ksoftirqd/223:52:232
2601799243,7cyclictest27-21ksoftirqd/223:15:162
2601799243,6cyclictest27-21ksoftirqd/221:25:192
26017992414,4cyclictest0-21swapper/219:55:232
2601799241,17cyclictest26346irq/40-dwc2_hso23:40:122
2601799240,9cyclictest26038-21ls22:40:192
2600999243,14cyclictest27613-21ssh22:00:151
2600999242,17cyclictest842-21munin-node19:50:151
2600999242,17cyclictest842-21munin-node19:50:151
2600999241,20cyclictest0-21swapper/100:30:141
2600199244,13cyclictest25057-21ssh23:20:140
2600199243,6cyclictest53046usb-storage20:05:040
2600199242,17cyclictest565-21ssh23:25:120
26001992414,7cyclictest22763-21cron23:17:010
26001992412,4cyclictest131rcu_preempt23:40:010
26025992315,5cyclictest29566irq/58-eth022:15:153
2601799238,10cyclictest131rcu_preempt21:52:132
2601799236,13cyclictest261rcuc/223:45:152
2601799235,8cyclictest27-21ksoftirqd/220:35:142
2601799233,10cyclictest27-21ksoftirqd/200:20:012
2601799232,10cyclictest9270-21fschecks_time23:10:182
26017992316,4cyclictest31562-21ssh21:40:242
26017992312,9cyclictest0-21swapper/222:16:082
2601799230,20cyclictest23884-21ls20:45:242
2600999232,17cyclictest24099-21apt-get20:15:141
2600199238,12cyclictest4350irq/33-fe00b88000:30:280
26001992312,4cyclictest0-21swapper/021:05:140
2600199231,17cyclictest7081-21ls19:20:280
2601799227,6cyclictest131rcu_preempt22:58:452
2601799224,15cyclictest893-21latency_hist23:05:022
2601799223,12cyclictest8569-21ssh23:30:122
2601799223,12cyclictest26346irq/40-dwc2_hso20:25:142
2601799221,6cyclictest0-21swapper/219:35:132
26017992212,8cyclictest0-21swapper/222:00:172
26017992212,4cyclictest0-21swapper/221:05:182
2601799221,11cyclictest30017-21idleruntime-cro20:55:022
2601799221,10cyclictest10034-21rm22:30:152
26017992210,9cyclictest0-21swapper/221:55:132
26017992210,3cyclictest27-21ksoftirqd/222:25:172
26017992210,3cyclictest27-21ksoftirqd/222:25:162
2600999227,8cyclictest0-21swapper/122:55:141
2600999221,18cyclictest7564-21find21:05:141
2600999221,18cyclictest28450-21find21:40:131
2600999221,16cyclictest19565-21ssh21:55:131
2600199223,6cyclictest0-21swapper/020:20:140
26001992210,9cyclictest19573-21diskmemload21:55:120
2602599213,11cyclictest0-21swapper/322:50:123
2601799218,6cyclictest1-21systemd00:08:212
2601799217,11cyclictest16521-21kthreadcore20:05:212
2601799213,13cyclictest26346irq/40-dwc2_hso21:45:142
2601799211,3cyclictest131rcu_preempt21:37:292
26017992112,7cyclictest0-21swapper/221:18:582
26017992112,7cyclictest0-21swapper/200:20:152
2601799211,10cyclictest16042-21kthreadcore00:35:202
2601799211,10cyclictest13057-21irqrtprio21:30:202
26017992110,4cyclictest0-21swapper/223:59:292
2600999217,8cyclictest842-21munin-node22:35:141
2600999214,11cyclictest0-21swapper/120:40:171
2600999212,15cyclictest14182-21apt-get20:05:041
2600999212,14cyclictest25556-21ssh23:00:141
2600999212,13cyclictest16287-21ssh23:55:121
2600999211,17cyclictest25539-21cut20:50:181
2600999211,17cyclictest25221-21cut23:20:171
2600999211,15cyclictest4710-21ssh23:05:261
26001992114,5cyclictest0-21swapper/023:40:290
26001992113,3cyclictest12-21ksoftirqd/019:35:140
2600199211,17cyclictest27678-21cut19:45:180
2600199211,15cyclictest6297-21cut20:30:190
2602599209,8cyclictest0-21swapper/322:03:323
26025992012,6cyclictest0-21swapper/322:21:573
26025992011,7cyclictest0-21swapper/322:34:503
26025992011,7cyclictest0-21swapper/321:12:053
26025992011,6cyclictest0-21swapper/323:10:293
2602599201,14cyclictest0-21swapper/319:15:153
2601799205,12cyclictest8391-21grep19:25:172
2601799205,12cyclictest8391-21grep19:25:172
2601799205,12cyclictest26440-21idleruntime19:10:182
2601799203,9cyclictest1056-21/usr/sbin/munin23:05:172
2601799201,9cyclictest25672-21kthreadcore20:15:212
2601799201,9cyclictest15989-21idleruntime20:40:202
26017992011,7cyclictest0-21swapper/219:40:212
2600999203,9cyclictest25993-21cyclictest19:20:121
2600999203,3cyclictest131rcu_preempt22:30:121
2600999201,9cyclictest1662-21kthreadcore00:05:221
2600999201,16cyclictest21085-21kthreadcore19:35:221
2600999201,16cyclictest20642-21sh21:15:151
2600999201,15cyclictest31129-21cut19:15:161
2600999201,15cyclictest14906-21ssh21:10:241
2600199202,3cyclictest131rcu_preempt21:10:160
2600199202,12cyclictest16309-21sh23:55:130
2600199202,12cyclictest11046-21ssh22:10:170
2600199201,2cyclictest0-21swapper/023:31:490
2600199201,16cyclictest8414-21cut00:10:170
2600199201,15cyclictest28489-21sh21:40:160
2600199201,14cyclictest29079-21ssh22:40:230
26001992010,7cyclictest135-21systemd-journal21:00:010
2602599198,6cyclictest0-21swapper/322:06:233
2602599192,11cyclictest0-21swapper/300:15:163
2602599192,10cyclictest0-21swapper/320:05:153
26025991911,6cyclictest0-21swapper/322:55:163
2602599191,13cyclictest0-21swapper/300:00:133
2602599191,12cyclictest0-21swapper/321:20:113
2602599191,12cyclictest0-21swapper/320:15:293
26025991910,6cyclictest0-21swapper/323:35:453
26025991910,6cyclictest0-21swapper/300:24:283
2600999199,7cyclictest4951-21ntpq22:45:261
2600999196,10cyclictest24655-21cron23:20:021
2600999192,14cyclictest23143-21kworker/1:023:40:141
2600999191,9cyclictest15898-21sshd22:10:431
2600999191,16cyclictest5419-21sed19:55:191
26009991911,5cyclictest9609-21fschecks_time22:50:151
2600199192,15cyclictest1989-21kthreadcore23:45:210
2600199191,7cyclictest4607-21irqstats21:45:190
2600199191,6cyclictest167-21systemd-network22:48:290
2600199191,15cyclictest25438-21cat22:40:010
2600199191,15cyclictest19571-21ls20:10:220
2600199191,13cyclictest7866-21ssh22:05:330
2602599189,7cyclictest0-21swapper/300:30:253
2602599188,6cyclictest0-21swapper/322:46:073
2602599185,10cyclictest33-21ksoftirqd/321:50:143
2602599183,12cyclictest0-21swapper/321:45:133
2602599182,11cyclictest0-21swapper/323:45:143
2602599182,10cyclictest0-21swapper/319:35:153
26025991812,4cyclictest33-21ksoftirqd/321:25:213
2602599181,12cyclictest0-21swapper/319:40:303
2602599181,11cyclictest0-21swapper/320:35:233
2602599181,10cyclictest0-21swapper/319:10:033
26025991810,6cyclictest0-21swapper/322:35:203
26025991810,6cyclictest0-21swapper/321:19:493
2600999189,7cyclictest28944-21apt-get20:20:131
2600999189,7cyclictest16672-21sed00:15:191
2600999188,7cyclictest8283-21apt-get00:10:141
2600999185,10cyclictest13392-21sed19:30:191
2600999184,11cyclictest26269-21irqcore22:40:191
2600999181,5cyclictest12275-21grep21:50:191
26009991812,4cyclictest0-21swapper/119:25:241
26009991812,4cyclictest0-21swapper/119:25:241
2600999181,15cyclictest11113-21fschecks_time20:35:171
2600999181,14cyclictest8215-21munin-plugin-st23:30:011
2600999181,14cyclictest1332-21apt-get20:25:131
2600199183,4cyclictest131rcu_preempt23:00:000
2600199183,12cyclictest26348-21munin-run22:20:010
2600199183,11cyclictest0-21swapper/020:40:180
2600199182,5cyclictest131rcu_preempt00:25:010
2600199181,15cyclictest22814-21uniq19:40:180
2600199181,13cyclictest28722-21idleruntime-cro20:20:010
2602599179,6cyclictest0-21swapper/323:15:113
2602599179,6cyclictest0-21swapper/300:36:593
2602599179,6cyclictest0-21swapper/300:11:503
2602599179,5cyclictest29566irq/58-eth019:55:143
2602599179,3cyclictest0-21swapper/323:31:513
2602599172,9cyclictest0-21swapper/320:45:163
2602599172,10cyclictest0-21swapper/322:40:153
2602599172,10cyclictest0-21swapper/321:05:153
2602599172,10cyclictest0-21swapper/319:10:143
26025991712,3cyclictest33-21ksoftirqd/319:45:293
2602599171,13cyclictest0-21swapper/320:20:243
2602599171,13cyclictest0-21swapper/300:25:213
2602599171,12cyclictest0-21swapper/320:50:183
2602599171,12cyclictest0-21swapper/320:45:023
2600999176,8cyclictest21606-21cron00:40:011
26009991711,4cyclictest26393-21timerandwakeup19:40:301
2600999171,14cyclictest19263-21kthreadcore23:35:231
2600999171,12cyclictest27459-21kthreadcore22:20:201
26009991710,4cyclictest4451-21cpu21:25:161
2600999170,8cyclictest6151-21cpu20:30:161
2600199171,8cyclictest28950-21kthreadcore19:10:210
26001991712,3cyclictest0-21swapper/022:26:230
26001991712,3cyclictest0-21swapper/022:26:230
26001991710,5cyclictest4428-21tr21:00:220
2600199170,12cyclictest24315-21ls20:45:300
2600199170,11cyclictest4516-21latency_hist19:55:020
2600199170,11cyclictest4516-21latency_hist19:55:010
2602599161,10cyclictest0-21swapper/323:40:053
2600999168,5cyclictest31321-21kthreadcore20:55:221
2600999168,5cyclictest28024-21irqrtprio19:45:191
2600999165,9cyclictest19651-21tail20:10:221
2600999162,11cyclictest2956-21hwlatdetect21:00:181
2600999162,11cyclictest21522-21kthreadcore21:35:201
2600999161,12cyclictest26660-21sed19:10:181
2600999160,13cyclictest23168-21kthreadcore20:45:241
2600199168,6cyclictest9031-21apt-get23:10:140
2600199165,2cyclictest131rcu_preempt19:55:180
2600199161,12cyclictest24490-21apt-get00:00:130
2600199161,11cyclictest32437-21irqrtprio00:25:220
2602599151,11cyclictest0-21swapper/320:55:173
26025991510,3cyclictest33-21ksoftirqd/319:25:193
26025991510,3cyclictest33-21ksoftirqd/319:25:183
2600199159,4cyclictest26003-21ls19:10:010
26025991410,2cyclictest33-21ksoftirqd/321:00:243
2601799141,10cyclictest26037-21latency_hist19:10:022
2600999141,10cyclictest26042-21grep19:10:031
2602599137,4cyclictest33-21ksoftirqd/319:20:173
2602599135,5cyclictest29566irq/58-eth020:10:183
2602599133,7cyclictest0-21swapper/320:00:163
2602599121,9cyclictest0-21swapper/320:30:223
2602599121,9cyclictest0-21swapper/319:50:273
2602599121,9cyclictest0-21swapper/319:50:263
2602599121,5cyclictest0-21swapper/320:25:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional