You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 06:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Nov 27, 2025 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19189996237,17cyclictest29566irq/58-eth022:40:123
1917299582,47cyclictest32298-21ssh21:50:121
1918999562,38cyclictest29566irq/58-eth023:05:133
1918199542,36cyclictest12429-21ssh22:55:132
1917299532,40cyclictest16260-21ssh23:55:131
1916399511,8cyclictest0-21swapper/020:40:100
1917299504,38cyclictest29230-21ssh23:05:131
1918999494,28cyclictest29566irq/58-eth000:30:133
1917299465,35cyclictest13539-21rm23:15:121
1917299461,6cyclictest131rcu_preempt21:05:121
19163994635,5cyclictest131rcu_preempt19:30:120
1917299454,34cyclictest6526-21ssh23:30:141
1917299441,35cyclictest23360-21ssh23:40:111
1917299441,35cyclictest23360-21ssh23:40:111
1917299431,32cyclictest25220-21ssh22:05:121
1917299431,32cyclictest25220-21ssh22:05:121
1916399432,38cyclictest19465-21rm22:40:110
19189994221,13cyclictest29566irq/58-eth020:10:143
19189994221,13cyclictest29566irq/58-eth020:10:143
19181994215,14cyclictest19155-21cyclictest21:15:122
1916399415,11cyclictest131rcu_preempt23:45:120
1916399415,11cyclictest131rcu_preempt23:45:120
1916399411,34cyclictest16255-21ssh23:55:120
1918999403,22cyclictest29566irq/58-eth021:20:123
1918199402,20cyclictest22740-21cut23:00:202
1916399404,3cyclictest131rcu_preempt00:00:140
1916399397,26cyclictest7172-21ssh00:30:120
1916399393,31cyclictest18090-21ssh22:20:140
1918199382,28cyclictest24674-21cut21:25:202
1918199382,18cyclictest26717-21grep22:05:202
1918199382,18cyclictest26717-21grep22:05:202
1917299383,28cyclictest15327-21ssh21:40:111
1917299383,28cyclictest15327-21ssh21:40:111
1916399381,30cyclictest26463-21ssh22:25:120
1918999373,20cyclictest29566irq/58-eth021:45:123
1918199372,18cyclictest2224-21grep21:50:202
1918199363,22cyclictest0-21swapper/221:10:142
1918199362,22cyclictest19467-21ssh22:40:112
1918199352,27cyclictest21008-21ssh00:38:512
1918199352,27cyclictest21008-21ssh00:38:512
1918199352,20cyclictest23349-21ssh23:40:102
1918199352,20cyclictest23349-21ssh23:40:102
1918999342,20cyclictest29566irq/58-eth000:00:133
1918199342,23cyclictest0-21swapper/222:03:202
19163993419,9cyclictest12-21ksoftirqd/022:47:010
1916399340,24cyclictest1262-21sh22:10:140
1918199333,22cyclictest629-21Async14:00:482
1918199333,22cyclictest629-21Async14:00:482
1917299332,22cyclictest13203-21kthreadcore22:35:211
19163993314,11cyclictest12-21ksoftirqd/020:55:110
1918199323,17cyclictest23833-21ssh21:45:132
1918199322,17cyclictest16271-21ssh23:55:132
19172993220,5cyclictest0-21swapper/119:50:121
1918199313,9cyclictest0-21swapper/221:20:202
19181993117,9cyclictest27-21ksoftirqd/223:50:172
1917299312,23cyclictest30011-21ssh21:47:201
1917299312,22cyclictest27108-21ssh22:44:201
1916399312,23cyclictest14158-21ssh21:56:520
1918199303,21cyclictest0-21swapper/223:22:202
1918199302,16cyclictest26346irq/40-dwc2_hso20:30:082
1918199302,11cyclictest537-21cut21:30:202
19181993014,10cyclictest2460-21ssh22:30:152
1917299302,20cyclictest24183-21df19:15:141
19172993014,7cyclictest19155-21cyclictest21:00:131
19172993014,7cyclictest19155-21cyclictest21:00:131
1918999293,12cyclictest29566irq/58-eth000:35:133
1918999293,12cyclictest29566irq/58-eth000:35:133
1918199293,16cyclictest26346irq/40-dwc2_hso20:20:112
1918199293,15cyclictest26346irq/40-dwc2_hso00:25:142
1918199292,20cyclictest0-21swapper/220:50:132
1918199292,20cyclictest0-21swapper/219:10:142
1917299292,21cyclictest24652-21ssh00:00:131
19172992919,6cyclictest0-21swapper/123:25:151
19172992911,12cyclictest30682-21ssh21:30:121
1918999287,15cyclictest29566irq/58-eth021:50:133
1918999283,12cyclictest29566irq/58-eth022:30:153
19189992816,8cyclictest0-21swapper/323:40:103
19189992816,8cyclictest0-21swapper/323:40:103
1918199283,17cyclictest26346irq/40-dwc2_hso19:20:162
1918199282,19cyclictest0-21swapper/220:40:102
1918199282,10cyclictest18600-21ssh22:39:202
19181992816,6cyclictest0-21swapper/222:25:122
1918199281,18cyclictest19551-21ls21:00:022
1917299282,20cyclictest16794-21cut21:20:201
1917299282,19cyclictest629-21Async14:00:481
1917299282,18cyclictest22738-21kthreadcore23:00:201
1917299281,21cyclictest0-21swapper/100:20:131
1916399283,14cyclictest30686-21ssh21:30:120
1916399282,19cyclictest18594-21tr21:40:210
1916399282,19cyclictest18594-21tr21:40:210
1916399282,18cyclictest13011-21kthreadcore22:35:210
1918999273,15cyclictest0-21swapper/323:02:043
19189992711,10cyclictest29566irq/58-eth021:30:133
19189992710,11cyclictest0-21swapper/322:21:573
1918199272,16cyclictest26346irq/40-dwc2_hso00:05:122
1917299272,19cyclictest21009-21sshd00:38:511
1917299272,19cyclictest21009-21sshd00:38:511
1916399272,19cyclictest6057-21ssh21:52:200
1916399272,19cyclictest29026-21diskmemload22:07:200
1916399272,19cyclictest29026-21diskmemload22:07:200
1916399272,19cyclictest26777-21ssh23:02:040
1916399272,19cyclictest14381-21ssh21:39:200
1918999260,14cyclictest29566irq/58-eth019:20:143
1918199269,12cyclictest27-21ksoftirqd/222:47:012
1918199262,9cyclictest6745-21ssh21:35:132
1918199262,6cyclictest0-21swapper/219:15:122
1918199261,17cyclictest9654-21ssh22:15:112
1918199261,16cyclictest29009-21ls19:50:222
1918199261,14cyclictest73-21kswapd023:35:132
1916399265,16cyclictest4076-21ssh22:50:140
1916399262,2cyclictest131rcu_preempt19:10:140
1916399262,17cyclictest24654-21ssh21:25:200
19163992617,6cyclictest12-21ksoftirqd/022:55:130
1918999255,8cyclictest29566irq/58-eth023:45:123
1918999255,8cyclictest29566irq/58-eth023:45:123
1918999253,16cyclictest0-21swapper/321:40:213
1918999253,16cyclictest0-21swapper/321:40:213
1918999253,13cyclictest0-21swapper/323:30:213
1918199251,9cyclictest21567-21ssh21:42:202
1918199251,9cyclictest21567-21ssh21:42:202
19181992511,9cyclictest13248-21sh00:10:312
1918199251,17cyclictest26346irq/40-dwc2_hso23:25:162
1918199251,15cyclictest7182-21ssh00:30:132
1917299252,17cyclictest14955-21ssh23:35:131
1916399252,18cyclictest23664-21ssh22:03:200
1916399252,18cyclictest1-21systemd00:39:080
1916399252,18cyclictest1-21systemd00:39:070
1918999246,8cyclictest29566irq/58-eth022:15:113
1918999242,18cyclictest0-21swapper/321:15:123
1918999242,14cyclictest29566irq/58-eth000:20:123
19189992410,9cyclictest29566irq/58-eth022:10:153
1918199249,11cyclictest27-21ksoftirqd/222:21:572
1918199242,15cyclictest26346irq/40-dwc2_hso20:10:152
1918199242,15cyclictest26346irq/40-dwc2_hso20:10:152
19181992414,8cyclictest0-21swapper/200:00:162
1918199241,18cyclictest5138-21ssh23:10:122
1917299246,13cyclictest30232-1kworker/u9:221:36:481
1917299241,17cyclictest7942-21sh23:50:171
1916399244,17cyclictest8433-21ssh00:10:120
1916399242,12cyclictest12-21ksoftirqd/020:15:140
19163992418,4cyclictest0-21swapper/021:45:130
1916399241,3cyclictest131rcu_preempt23:10:130
1918999232,14cyclictest0-21swapper/321:27:203
19189992311,8cyclictest29566irq/58-eth022:50:133
1918199239,2cyclictest27-21ksoftirqd/219:25:282
1918199238,6cyclictest12055-21cron22:55:012
1918199236,10cyclictest0-21swapper/219:30:122
1918199232,16cyclictest0-21swapper/221:58:202
1918199232,11cyclictest31787-21ssh23:45:122
1918199232,11cyclictest31787-21ssh23:45:122
1918199231,17cyclictest29810-21cut23:05:172
1917299234,13cyclictest842-21munin-node23:10:121
19172992313,6cyclictest29026-21diskmemload22:45:151
1917299231,17cyclictest29413-21ssh21:10:141
1917299231,16cyclictest8454-21ssh00:10:141
19163992314,7cyclictest0-21swapper/023:15:210
19163992314,6cyclictest16408-21ssh00:15:130
1918999223,13cyclictest0-21swapper/322:25:123
1918999223,13cyclictest0-21swapper/321:55:203
1918199224,9cyclictest25860-21kthreadcore21:05:192
1918199223,16cyclictest14582-21cron23:35:012
1918199221,8cyclictest0-21swapper/200:20:132
19181992210,10cyclictest12465-21sshd20:06:202
1916399222,15cyclictest29026-21diskmemload23:22:080
1918999219,9cyclictest0-21swapper/322:05:183
1918999219,9cyclictest0-21swapper/322:05:183
1918999213,13cyclictest0-21swapper/321:35:203
1918199216,11cyclictest5218-21kthreadcore20:00:222
1918199215,7cyclictest21166-21grep20:15:282
19181992112,7cyclictest0-21swapper/221:00:142
19181992112,7cyclictest0-21swapper/221:00:142
19181992111,6cyclictest27-21ksoftirqd/223:16:172
1918199211,10cyclictest932-21snmpd19:10:022
1917299211,7cyclictest1955-21irqstats22:10:201
1917299211,17cyclictest10195-21tr20:50:171
1917299210,18cyclictest1270-21ls19:25:021
1916399213,11cyclictest151rcuc/021:20:110
1916399212,14cyclictest1-21systemd19:15:130
1916399211,5cyclictest29036-21munin-run21:10:010
19163992113,5cyclictest0-21swapper/023:40:240
19163992113,5cyclictest0-21swapper/023:40:240
19163992112,7cyclictest9950-21ssh22:15:160
1916399211,15cyclictest5511-21uniq21:15:150
1918999208,10cyclictest0-21swapper/300:16:513
1918999204,12cyclictest29566irq/58-eth000:25:153
19189992013,5cyclictest0-21swapper/323:35:163
1918199209,8cyclictest0-21swapper/222:10:022
1918199203,9cyclictest1807-21switchtime19:55:292
19181992011,7cyclictest0-21swapper/200:17:302
1918199201,10cyclictest17223-21kthreadcore19:40:192
1918199201,10cyclictest15959-21latency_hist19:40:022
1917299209,8cyclictest12417-21apt-get22:55:131
1917299203,10cyclictest0-21swapper/121:55:201
19163992012,6cyclictest19359-21ssh23:35:290
19163992011,6cyclictest7379-21ssh22:30:320
1916399201,14cyclictest28986-21sshd19:20:150
1916399201,11cyclictest8081-21latency_hist00:10:020
1916399200,15cyclictest13183-21ls20:50:210
1918999198,8cyclictest0-21swapper/323:25:243
1918999196,7cyclictest0-21swapper/323:15:123
19189991911,5cyclictest29566irq/58-eth022:55:143
1918999191,11cyclictest0-21swapper/319:15:013
1918199191,9cyclictest32353-21sh20:40:002
1918199191,14cyclictest5126-21fschecks_count20:45:152
1917299198,8cyclictest5111-21kthreadcore22:30:231
1917299196,5cyclictest4283-21fschecks_time22:50:171
1917299192,11cyclictest29829-21ssh21:29:201
1917299191,9cyclictest0-21swapper/100:32:091
19172991913,4cyclictest20618-21sh22:20:211
19172991911,5cyclictest20686-21switchtime22:00:291
19172991911,5cyclictest0-21swapper/119:30:251
1917299191,13cyclictest32400-21ssh00:25:151
1917299191,12cyclictest21961-21ssh23:20:111
1917299191,11cyclictest11230-21ls20:05:231
1916399196,3cyclictest1570-21/usr/sbin/munin19:55:250
1916399196,10cyclictest29464-21awk21:10:140
1916399192,3cyclictest131rcu_preempt00:20:140
19163991911,5cyclictest13001-21ssh23:32:480
1916399191,14cyclictest7928-21ssh23:50:160
19163991910,7cyclictest16219-21ntp_states20:10:270
19163991910,7cyclictest16219-21ntp_states20:10:270
19163991910,7cyclictest0-21swapper/023:06:120
19163991910,6cyclictest2032-21kthreadcore00:25:220
1916399190,12cyclictest22558-21cut20:20:140
1918999183,12cyclictest0-21swapper/322:00:243
1918999183,10cyclictest29566irq/58-eth022:45:143
1918999183,10cyclictest0-21swapper/323:22:083
1918999182,11cyclictest0-21swapper/321:14:203
1918999182,10cyclictest0-21swapper/320:45:143
1918999182,10cyclictest0-21swapper/320:30:093
1918999182,10cyclictest0-21swapper/319:55:163
1918999182,10cyclictest0-21swapper/319:45:133
1918999182,10cyclictest0-21swapper/319:45:123
1918999182,10cyclictest0-21swapper/319:35:153
1918999181,9cyclictest0-21swapper/320:50:123
1918999181,13cyclictest0-21swapper/319:40:203
1918999181,12cyclictest0-21swapper/321:00:183
1918999181,12cyclictest0-21swapper/321:00:183
1918999181,10cyclictest0-21swapper/319:25:163
1918199181,14cyclictest27264-21cron20:30:012
1917299189,6cyclictest19488-21wc19:10:161
1917299188,8cyclictest21233-21users00:15:321
1917299188,7cyclictest18633-21grep20:55:271
1917299182,10cyclictest0-21swapper/119:40:141
19172991810,5cyclictest10020-21irqcore22:15:151
19172991810,4cyclictest2210-21grep22:30:011
1917299180,15cyclictest24194-21ls19:45:211
1917299180,15cyclictest24194-21ls19:45:211
1916399182,11cyclictest0-21swapper/019:45:130
1916399182,11cyclictest0-21swapper/019:45:130
1916399181,14cyclictest5503-21ps20:45:180
19163991811,4cyclictest32350-21munin-run20:40:010
1918999179,6cyclictest0-21swapper/323:10:113
1918999179,6cyclictest0-21swapper/300:12:343
1918999179,3cyclictest0-21swapper/323:55:063
1918999173,8cyclictest0-21swapper/322:35:213
1918999172,9cyclictest0-21swapper/321:05:133
1918999172,9cyclictest0-21swapper/320:20:123
1918999172,10cyclictest0-21swapper/319:30:133
19189991711,4cyclictest0-21swapper/300:05:323
1918999171,12cyclictest0-21swapper/323:50:323
1917299178,6cyclictest13437-21cut20:10:201
1917299178,6cyclictest13437-21cut20:10:201
1917299176,8cyclictest22772-21grep20:20:161
1917299175,9cyclictest31069-21irqrtprio19:55:201
1917299173,11cyclictest14972-21grep19:35:291
1917299172,5cyclictest32596-21apt-get20:40:101
1917299171,8cyclictest20964-21missed_timers20:15:241
1917299171,13cyclictest6996-21kthreadcore20:45:201
1917299171,12cyclictest8974-21ntp_states21:15:251
1916399175,9cyclictest4792-21kthreadcore20:00:220
1916399171,4cyclictest131rcu_preempt23:25:260
1916399171,13cyclictest8006-21cut20:05:160
1918999162,10cyclictest0-21swapper/319:15:133
1917299167,7cyclictest31775-21apt-get23:45:111
1917299167,7cyclictest31775-21apt-get23:45:111
1917299166,7cyclictest3017-21apt-get20:00:141
1917299161,12cyclictest28907-21kthreadcore20:30:151
19172991610,3cyclictest1778-21cut19:25:171
1916399168,6cyclictest13203-21cut19:35:210
1916399162,9cyclictest0-21swapper/019:40:140
1916399162,11cyclictest19168-21latency_hist19:10:010
1916399162,10cyclictest0-21swapper/021:00:140
1916399162,10cyclictest0-21swapper/021:00:140
1916399162,10cyclictest0-21swapper/019:25:130
1916399161,13cyclictest26570-21kthreadcore19:50:190
1916399161,12cyclictest27595-21diskstats20:30:110
1918999152,9cyclictest0-21swapper/320:40:103
1918999152,9cyclictest0-21swapper/320:15:153
1917299151,11cyclictest32465-21latency_hist20:40:011
1917299151,10cyclictest19200-21latency_hist19:10:021
1916399151,3cyclictest0-21swapper/020:28:190
1918999149,3cyclictest0-21swapper/320:05:273
1918999141,10cyclictest0-21swapper/320:05:023
1917299141,10cyclictest27265-21cron20:30:011
1918999135,5cyclictest33-21ksoftirqd/320:55:163
1918999131,10cyclictest33-21ksoftirqd/319:50:273
1918999128,2cyclictest33-21ksoftirqd/320:27:303
1918999127,3cyclictest0-21swapper/320:35:193
1918999101,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional