You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-24 - 17:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Wed Jun 24, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1177329574,6sleep00-21swapper/007:09:060
1182728875,9sleep20-21swapper/207:09:482
1173728261,6sleep10-21swapper/107:08:391
1170428262,9sleep30-21swapper/307:08:143
155562630,0sleep10-21swapper/110:32:221
11931992826,1cyclictest0-21swapper/111:13:421
1193199251,3cyclictest141rcu_preempt11:40:261
1193199230,1cyclictest0-21swapper/109:43:451
1193199230,1cyclictest0-21swapper/109:43:441
11931992220,1cyclictest0-21swapper/110:25:171
1193199211,2cyclictest141rcu_preempt11:32:371
1193199210,1cyclictest0-21swapper/112:05:141
1193199210,18cyclictest0-21swapper/112:15:121
11926992118,2cyclictest13-21ksoftirqd/010:53:130
116712210,0sleep10-21swapper/109:13:441
1193399200,17cyclictest0-21swapper/209:45:142
11931992018,1cyclictest25019-21ssh10:00:121
1193199200,18cyclictest0-21swapper/110:42:291
207512190,0sleep10-21swapper/112:30:361
1193699199,9cyclictest4938-21apache210:08:093
1193399190,19cyclictest0-21swapper/209:14:302
1193399190,18cyclictest0-21swapper/212:38:262
1193399190,18cyclictest0-21swapper/209:23:022
1193199190,2cyclictest0-21swapper/111:56:381
1193199190,15cyclictest0-21swapper/109:34:201
1192699196,6cyclictest25234-21awk12:15:010
1192699196,6cyclictest25234-21awk12:15:000
242062180,1chrt24207-21sh09:20:273
1193199180,17cyclictest0-21swapper/109:38:521
1193199180,15cyclictest0-21swapper/110:09:181
1192699185,7cyclictest30063-21munin-run09:25:000
1192699182,14cyclictest29935-21apt-get08:55:030
11926991817,1cyclictest0-21swapper/011:50:480
11926991817,1cyclictest0-21swapper/010:01:190
1192699181,1cyclictest141rcu_preempt11:17:440
11936991716,1cyclictest0-21swapper/309:57:003
11936991713,2cyclictest0-21swapper/312:29:133
1193399176,1cyclictest141rcu_preempt12:11:282
1193399176,1cyclictest141rcu_preempt12:11:272
1193399171,1cyclictest22061-21cpu11:15:152
1193399170,17cyclictest0-21swapper/211:52:432
1193399170,17cyclictest0-21swapper/211:11:192
1193399170,16cyclictest0-21swapper/210:31:092
1193199173,1cyclictest3939-21ssh10:45:131
11931991716,1cyclictest0-21swapper/110:59:341
1193199171,1cyclictest0-21swapper/109:26:031
1193199170,17cyclictest0-21swapper/112:04:131
1193199170,16cyclictest4873-21diskmemload11:53:371
1192699170,1cyclictest0-21swapper/012:02:580
1193699162,5cyclictest273-21systemd-udevd12:34:033
11936991615,1cyclictest0-21swapper/311:11:233
1193699160,14cyclictest0-21swapper/312:22:593
1193399160,13cyclictest0-21swapper/211:47:392
1193199163,7cyclictest0-21swapper/108:45:131
11931991614,1cyclictest15264-21sed08:20:251
11931991614,1cyclictest0-21swapper/111:49:391
1193199160,13cyclictest0-21swapper/111:09:401
1192699160,1cyclictest0-21swapper/010:11:000
1193699154,5cyclictest225-21systemd-journal08:20:013
11936991512,2cyclictest19135-21ls08:30:173
11936991511,3cyclictest17763-21which11:50:203
1193699150,3cyclictest0-21swapper/311:16:313
1193399159,2cyclictest27324-21cron10:40:012
11931991515,0cyclictest0-21swapper/111:26:381
11931991513,1cyclictest4873-21diskmemload10:38:371
1192699150,13cyclictest0-21swapper/008:00:130
1193699145,8cyclictest3489-21sh08:00:003
11933991411,2cyclictest24099-21df09:40:152
11933991411,2cyclictest24099-21df09:40:142
1193399140,3cyclictest0-21swapper/210:59:222
1193399140,14cyclictest0-21swapper/210:22:592
1193399140,14cyclictest0-21swapper/209:53:302
1193399140,13cyclictest0-21swapper/210:41:342
1193199147,4cyclictest30259-21apt-get11:20:121
11931991410,3cyclictest8821-21grep09:50:171
11931991410,2cyclictest1957-21tr12:20:141
1193199140,13cyclictest12583-21wc08:15:251
1193199140,13cyclictest0-21swapper/109:21:161
1193199140,12cyclictest0-21swapper/112:10:131
1193199140,12cyclictest0-21swapper/112:10:121
1192699146,7cyclictest32496-21munin-run12:00:000
1192699141,7cyclictest1208-21sendmail-msp12:20:010
1192699141,1cyclictest0-21swapper/009:41:500
1192699141,1cyclictest0-21swapper/009:41:490
11926991411,2cyclictest3358-21uniq10:25:150
1192699140,8cyclictest12981-21basename11:10:010
1193699137,5cyclictest4947-21apache209:54:233
11936991311,2cyclictest0-21swapper/309:34:103
11936991310,2cyclictest8103-21ssh11:25:253
11936991310,2cyclictest27486-21grep10:20:163
11936991310,2cyclictest23702-21cut08:40:153
11936991310,2cyclictest22180-21tr07:30:143
11936991310,2cyclictest20079-21ssh09:37:413
11936991310,2cyclictest19232-21strings10:15:203
1193699130,2cyclictest0-21swapper/312:09:333
1193699130,2cyclictest0-21swapper/309:15:373
1193399139,3cyclictest8483-21cron12:05:002
1193399139,3cyclictest13264-21head07:10:202
1193399137,5cyclictest16853-21sed07:20:002
11933991312,1cyclictest0-21swapper/211:40:372
11933991312,1cyclictest0-21swapper/208:15:142
1193399131,1cyclictest0-21swapper/211:09:512
11933991310,2cyclictest9134-21cron10:10:012
11933991310,2cyclictest27725-21wc07:40:202
11933991310,2cyclictest24765-21grep07:35:152
11933991310,2cyclictest1431-21cut07:55:102
1193399130,12cyclictest11386-21head12:25:202
11931991310,2cyclictest27748-21grep12:35:151
1192699139,3cyclictest25969-21grep07:35:270
1192699139,3cyclictest19103-21cat10:15:180
1192699134,3cyclictest16695-21sendmail-msp07:20:000
1192699131,6cyclictest28785-21munin-run07:45:000
11926991313,0cyclictest0-21swapper/010:57:500
1192699131,11cyclictest11733-21ls10:50:000
11926991310,2cyclictest7572-21wc09:50:010
11926991310,2cyclictest627-21wc11:40:230
11926991310,2cyclictest29142-21grep07:45:130
11926991310,2cyclictest26585-21tr08:45:200
11926991310,2cyclictest24540-21cat10:00:010
11926991310,2cyclictest19324-21cat07:25:000
1193699129,2cyclictest5094-21cut10:45:193
1193699129,2cyclictest3989-21grep09:05:223
1193699129,2cyclictest31446-21cut07:50:133
1193699129,2cyclictest31446-21cut07:50:123
1193699129,2cyclictest31276-21cut09:25:143
1193699129,2cyclictest28676-21head10:40:163
1193699129,2cyclictest27649-21tune2fs12:35:153
1193699129,2cyclictest25514-21sh10:00:143
1193699129,2cyclictest21392-21grep10:55:143
1193699129,2cyclictest1883-21ssh12:00:213
1193699129,2cyclictest17587-21sed08:25:283
1193699126,3cyclictest30259-21apt-get11:20:133
1193699126,2cyclictest225-21systemd-journal08:10:003
1193699124,6cyclictest1152-21cron12:20:013
1193699122,9cyclictest24276-21apt-get07:35:133
1193699121,8cyclictest0-21swapper/311:33:563
1193699120,11cyclictest0-21swapper/311:09:063
1193399129,2cyclictest9833-21head12:05:182
1193399129,2cyclictest29774-21grep07:45:162
1193399129,2cyclictest29374-21grep11:00:162
1193399129,2cyclictest26590-21head11:55:252
1193399129,2cyclictest24640-21head11:35:232
1193399129,2cyclictest24289-21basename10:00:002
1193399129,2cyclictest21393-21grep08:35:182
1193399129,2cyclictest1691-21apt-get12:20:122
1193399129,2cyclictest14576-21cut09:15:172
1193399128,3cyclictest31664-21tr07:50:142
1193399128,3cyclictest31664-21tr07:50:132
11933991210,1cyclictest26503-21grep12:15:162
1193399120,11cyclictest4379-21sed09:05:262
1193399120,11cyclictest10550-21cron10:29:592
1193199129,2cyclictest5638-21sed08:00:261
1193199129,2cyclictest32534-21ls08:59:591
1193199129,2cyclictest29196-21grep11:00:141
1193199129,2cyclictest22652-21cut07:30:151
1193199129,2cyclictest21779-21cat07:30:001
1193199129,2cyclictest19965-21grep08:30:271
1193199129,2cyclictest16490-21grep09:55:151
1193199129,2cyclictest15324-21head07:15:161
1193199124,4cyclictest225-21systemd-journal10:19:591
1193199123,5cyclictest1436-21sendmail-mta09:45:141
1192699129,2cyclictest9883-21grep08:10:190
1192699129,2cyclictest32011-21sort07:50:150
1192699129,2cyclictest32011-21sort07:50:150
1192699129,2cyclictest3151-21sed10:05:280
1192699129,2cyclictest23084-21awk09:40:010
1192699129,2cyclictest15255-21wc09:15:230
1192699129,2cyclictest14262-21grep11:10:160
1192699129,2cyclictest12835-21cut07:10:170
1192699129,2cyclictest11188-21grep10:30:150
1192699129,2cyclictest10293-21grep12:25:130
1192699128,3cyclictest16061-21grep11:30:200
1192699128,2cyclictest20595-21cat10:35:190
1192699121,10cyclictest2407-21dump-pmu-power12:40:000
1193699118,2cyclictest908-21cut09:45:213
1193699118,2cyclictest8319-21expr11:45:153
1193699118,2cyclictest3540-21ssh11:42:153
1193699118,2cyclictest3204-21grep10:25:153
1193699118,2cyclictest31850-21taskset11:59:313
1193699118,2cyclictest30649-21grep07:45:253
1193699118,2cyclictest29510-21sed08:50:273
1193699118,2cyclictest23689-21cut09:40:143
1193699118,2cyclictest23689-21cut09:40:133
1193699118,2cyclictest21835-21ssh10:35:453
1193699118,2cyclictest19590-21grep07:25:123
1193699118,2cyclictest18758-21grep12:10:203
1193699118,2cyclictest18758-21grep12:10:203
1193699118,2cyclictest15633-21cat07:15:183
1193699118,2cyclictest13634-21tr08:20:133
1193699118,2cyclictest12907-21grep07:10:173
1193699118,2cyclictest12423-21ntp_states10:30:223
1193699118,2cyclictest12260-21sh10:50:143
1193699118,2cyclictest10634-21taskset08:11:493
1193699115,2cyclictest225-21systemd-journal09:13:013
1193699114,5cyclictest9564-21/usr/sbin/munin10:10:213
1193699112,6cyclictest19323-21latency_hist07:25:013
1193699111,6cyclictest20826-21aten2_r7power_v08:35:153
11936991111,0cyclictest0-21swapper/311:01:583
1193699110,5cyclictest0-21swapper/311:35:013
1193699110,10cyclictest4363-21cat08:00:153
1193399118,2cyclictest4054-21cut10:45:132
1193399118,2cyclictest3203-21ssh11:23:042
1193399118,2cyclictest31645-21wc09:25:172
1193399118,2cyclictest24712-21cat10:00:012
1193399118,2cyclictest19954-21head10:15:262
1193399118,2cyclictest18128-21sed07:20:192
1193399118,2cyclictest16051-21grep11:30:202
1193399118,2cyclictest12182-21ssh10:50:122
1193399118,2cyclictest10427-21grep10:10:152
1193399118,2cyclictest10346-21ssh11:27:042
1193399117,3cyclictest24012-21grep07:35:012
1193399117,3cyclictest19810-21grep12:30:192
1193399117,3cyclictest17061-21wc09:35:292
1193399112,2cyclictest225-21systemd-journal08:10:422
1193199118,2cyclictest7724-21grep08:05:241
1193199118,2cyclictest32116-21grep07:50:161
1193199118,2cyclictest32116-21grep07:50:151
1193199118,2cyclictest3000-21tr09:05:151
1193199118,2cyclictest29795-21ssh11:39:001
1193199118,2cyclictest29783-21taskset08:53:131
1193199118,2cyclictest22384-21df11:15:161
1193199118,2cyclictest22017-21cat08:35:231
1193199118,2cyclictest20017-21ssh09:18:441
1193199118,2cyclictest1865-21wc09:00:231
1193199118,2cyclictest17480-21ssh10:15:001
1193199118,2cyclictest15625-21rm10:51:321
1193199118,2cyclictest13851-21head07:10:271
1193199118,2cyclictest10099-21ssh12:25:111
1193199110,10cyclictest26710-21cut10:20:121
1192699118,2cyclictest9059-21cut12:05:140
1192699118,2cyclictest8744-21cut09:50:160
1192699118,2cyclictest7782-21ssh09:10:410
1192699118,2cyclictest6908-21expr09:30:140
1192699118,2cyclictest3505-21basename08:00:010
1192699118,2cyclictest31548-21tr11:20:210
1192699118,2cyclictest30397-21ssh10:40:510
1192699118,2cyclictest30293-21chrt09:25:020
1192699118,2cyclictest28993-21ssh11:00:110
1192699118,2cyclictest20969-21tr08:35:150
1192699118,2cyclictest20435-21perf08:35:000
1192699118,2cyclictest19762-21cat12:30:200
1192699118,2cyclictest1757-21basename09:00:200
1192699118,2cyclictest16703-21sed08:25:180
1192699118,2cyclictest15689-21awk08:25:010
1192699118,2cyclictest10175-21ssh11:26:560
1192699117,3cyclictest1813-21grep12:20:130
1192699113,7cyclictest23275-21ssh11:35:130
1193699108,1cyclictest654-21expr09:00:133
1193699107,2cyclictest30665-21tr08:55:133
1193699107,2cyclictest27079-21grep08:45:263
1193399108,2cyclictest14159-21fschecks_count08:20:152
1193399108,1cyclictest64072sleep208:05:142
1193399108,1cyclictest5751-21head08:00:262
1193399108,1cyclictest295702sleep208:50:302
1193399108,1cyclictest27321-21chrt08:47:042
1193399108,1cyclictest19937-21expr07:25:132
1193399107,2cyclictest6583-21rm09:30:122
1193399107,2cyclictest24403-21expr08:40:222
1193399107,2cyclictest18535-21cut08:30:142
1193199107,2cyclictest23395-21cat08:40:141
1193199107,2cyclictest16558-21grep08:25:171
1193199106,2cyclictest25043-21cut07:35:181
1193199100,9cyclictest31033-21sshd07:48:421
1193199100,9cyclictest28859-21sed07:45:001
1193199100,9cyclictest10573-21taskset08:11:021
1192699108,1cyclictest7330-21irqstats08:05:210
1192699108,1cyclictest26233-21chmod10:20:010
1192699108,1cyclictest22568-21cut07:30:150
1192699108,1cyclictest21289-21chrt07:25:450
1192699107,2cyclictest8135-21tr11:45:140
1192699107,2cyclictest3960-21sed09:05:230
1192699107,2cyclictest23312-21grep08:40:140
1192699107,2cyclictest11349-21egrep08:15:140
119369996,2cyclictest27525-21if_enp2s007:40:183
119339997,1cyclictest2142-21taskset09:00:372
119339997,1cyclictest17763-21taskset08:26:312
119339996,3cyclictest32-21ksoftirqd/208:55:122
119319997,1cyclictest17706-21sed07:20:151
119319997,1cyclictest1721-21expr07:55:111
119269995,3cyclictest29050-21grep08:50:210
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional