You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 09:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Mar 25, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2621129776,16sleep00-21swapper/019:05:510
2647128575,6sleep20-21swapper/219:09:132
2620427966,9sleep30-21swapper/319:05:443
2626127666,6sleep10-21swapper/119:06:271
26630992727,0cyclictest0-21swapper/321:15:423
2662599267,12cyclictest0-21swapper/223:25:152
2663099250,25cyclictest0-21swapper/323:03:093
2662599246,10cyclictest0-21swapper/220:45:122
2662099242,4cyclictest141rcu_preempt21:38:290
26622992322,1cyclictest0-21swapper/120:50:031
2662299220,21cyclictest0-21swapper/123:37:141
26620992120,1cyclictest0-21swapper/023:31:290
2663099200,19cyclictest21619-21pmu-power23:25:223
26622992010,8cyclictest0-21swapper/121:22:501
2662299200,20cyclictest0-21swapper/121:16:541
2663099190,19cyclictest0-21swapper/323:24:013
26625991918,1cyclictest0-21swapper/221:21:022
2662299191,2cyclictest141rcu_preempt23:03:001
26620991919,0cyclictest0-21swapper/022:59:060
26620991918,1cyclictest0-21swapper/023:00:130
2663099184,2cyclictest141rcu_preempt23:37:013
26630991817,1cyclictest0-21swapper/321:08:333
2662299183,11cyclictest13655-21sh21:45:131
26622991813,4cyclictest0-21swapper/121:29:061
2662299180,18cyclictest0-21swapper/100:37:201
2662099184,7cyclictest21051-21munin-run23:45:010
2662099180,18cyclictest0-21swapper/022:47:320
26630991710,6cyclictest8484-21apache222:36:063
2662599172,14cyclictest0-21swapper/222:37:332
26625991715,2cyclictest32-21ksoftirqd/222:41:542
26625991714,1cyclictest0-21swapper/223:58:122
2662599170,16cyclictest0-21swapper/222:30:132
2662599170,16cyclictest0-21swapper/222:30:132
26622991714,2cyclictest25-21ksoftirqd/120:10:011
2662099170,1cyclictest0-21swapper/023:45:390
2662099170,13cyclictest0-21swapper/022:28:020
2663099160,15cyclictest19532-21fschecks_count23:05:163
2663099160,15cyclictest0-21swapper/323:46:043
2662299169,6cyclictest22947-21tr00:05:141
26622991614,1cyclictest0-21swapper/122:50:011
26622991613,2cyclictest0-21swapper/119:15:131
2662299161,14cyclictest19412-21diskmemload22:22:251
2662299160,3cyclictest0-21swapper/122:51:361
2662299160,16cyclictest0-21swapper/121:35:441
2662299160,16cyclictest0-21swapper/100:32:221
2662099163,7cyclictest32436-21awk22:35:000
2662099163,7cyclictest32436-21awk22:35:000
2662099161,8cyclictest225-21systemd-journal22:00:000
26620991616,0cyclictest0-21swapper/021:05:150
2662099160,1cyclictest0-21swapper/021:45:170
2662099160,16cyclictest0-21swapper/022:40:230
2662099160,16cyclictest0-21swapper/021:50:060
2662099160,0cyclictest0-21swapper/000:01:440
26625991514,1cyclictest0-21swapper/200:24:122
2662599151,1cyclictest141rcu_preempt23:19:582
26625991511,1cyclictest0-21swapper/200:34:012
26622991513,2cyclictest25-21ksoftirqd/122:31:391
26622991513,2cyclictest25-21ksoftirqd/122:31:391
26622991513,1cyclictest38150irq/131-snd_hda20:35:231
2662299151,13cyclictest0-21swapper/121:31:021
2662299150,14cyclictest19412-21diskmemload00:24:321
2662099152,6cyclictest9900-21munin-run20:50:010
2662099150,1cyclictest0-21swapper/023:50:480
2662099150,1cyclictest0-21swapper/021:34:500
2663099149,4cyclictest0-21swapper/300:28:253
26625991411,2cyclictest7648-21wc19:35:232
26625991411,2cyclictest1763-21grep20:30:282
2662599140,1cyclictest0-21swapper/200:11:262
2662599140,14cyclictest0-21swapper/223:23:522
2662299146,4cyclictest25-21ksoftirqd/122:00:321
26622991413,1cyclictest0-21swapper/121:13:461
2662099143,8cyclictest0-21swapper/023:35:150
26620991412,1cyclictest0-21swapper/020:40:130
26620991411,2cyclictest7450-21tr19:35:210
2662099140,7cyclictest0-21swapper/022:00:130
2662099140,14cyclictest0-21swapper/000:22:550
2662099140,13cyclictest21728-21cat20:05:190
2663099139,3cyclictest10403-21tr20:50:143
2663099134,8cyclictest0-21swapper/322:08:313
26630991310,2cyclictest31954-21sed23:50:253
26630991310,2cyclictest3118-21tr22:55:213
26630991310,2cyclictest31117-21cron22:15:013
26630991310,2cyclictest27091-21grep19:10:163
26630991310,2cyclictest23889-21grep21:50:263
26630991310,2cyclictest17592-21grep19:55:253
26630991310,2cyclictest15127-21chrt22:43:273
2663099130,12cyclictest32220-21chrt00:10:183
2662599139,3cyclictest8504-21tail22:20:162
2662599139,3cyclictest31321-21wc21:35:262
2662599131,10cyclictest9045-21taskset00:15:262
26625991310,2cyclictest29928-21cron20:25:002
26625991310,2cyclictest28529-21tr21:15:132
2662599130,12cyclictest0-21swapper/221:42:072
2662299137,5cyclictest31690-21ssh23:50:251
26622991311,1cyclictest19412-21diskmemload22:29:321
26622991310,2cyclictest7357-21wc19:35:201
26622991310,2cyclictest4188-21tr19:30:141
26622991310,2cyclictest32681-21wc19:20:191
26622991310,2cyclictest32178-21ls21:55:261
26622991310,2cyclictest23780-21grep20:10:191
26622991310,2cyclictest22823-21tail21:50:171
2662299130,1cyclictest141rcu_preempt23:45:301
2662299130,13cyclictest0-21swapper/122:07:011
2662099139,3cyclictest9916-21tail00:35:200
2662099139,2cyclictest27080-21tr19:10:150
26620991310,2cyclictest32388-21grep22:15:160
26620991310,2cyclictest23674-21grep20:10:190
26620991310,2cyclictest21460-21grep23:25:210
26620991310,2cyclictest19001-21tr20:00:170
26620991310,2cyclictest1579-21sed22:35:170
26620991310,2cyclictest13788-21tr19:50:140
26620991310,2cyclictest13450-21wc19:50:000
26620991310,2cyclictest12244-21awk21:24:590
2663099129,2cyclictest7359-21wc23:55:223
2663099129,2cyclictest32297-21cron20:30:003
2663099129,2cyclictest3025-21cat19:25:253
2663099129,2cyclictest25815-21tr22:30:223
2663099129,2cyclictest25815-21tr22:30:213
2663099129,2cyclictest24004-21cron22:30:003
2663099129,2cyclictest23441-21grep00:05:163
2663099129,2cyclictest19739-21ssh21:48:353
2663099129,2cyclictest1600-21ssh21:37:253
2663099121,3cyclictest0-21swapper/321:10:133
26630991210,1cyclictest14268-21head19:50:173
2663099120,2cyclictest0-21swapper/321:25:503
2663099120,11cyclictest15058-21head23:40:273
2662599129,2cyclictest6573-21sed23:35:282
2662599129,2cyclictest6254-21tr20:40:192
2662599129,2cyclictest30818-21head23:50:172
2662599129,2cyclictest30367-21tr19:15:202
2662599129,2cyclictest27931-21tail22:50:282
2662599129,2cyclictest27880-21wc19:10:202
2662599129,2cyclictest23324-21grep22:10:162
2662599129,2cyclictest2155-21tr22:55:152
2662599129,2cyclictest17737-21grep22:45:142
2662599129,2cyclictest17003-21head22:25:192
2662599129,2cyclictest16990-21grep19:55:182
2662599129,2cyclictest14735-21head21:25:272
2662599129,2cyclictest14491-21head21:45:182
2662599129,2cyclictest10498-21ssh23:00:132
2662599128,3cyclictest22281-21sed20:05:272
2662599128,3cyclictest21816-21cut23:45:142
2662599128,3cyclictest17291-21ssh00:01:092
2662599126,1cyclictest19935-21apt-get21:10:132
2662599122,7cyclictest0-21swapper/223:30:132
2662599120,12cyclictest0-21swapper/200:06:152
2662299129,2cyclictest32193-21grep00:10:181
2662299129,2cyclictest30906-21grep20:25:171
2662299129,2cyclictest27526-21grep23:10:141
2662299129,2cyclictest21906-21wc23:25:261
2662299129,2cyclictest15170-21sed00:00:151
2662299129,2cyclictest13920-21sed23:40:171
2662299129,2cyclictest13113-21cron21:44:591
2662299127,1cyclictest25-21ksoftirqd/123:15:471
2662299120,11cyclictest19514-21grep20:00:221
2662299120,11cyclictest14177-21cut19:50:161
2662299120,11cyclictest0-21swapper/119:55:211
2662099129,2cyclictest9429-21cut19:40:170
2662099129,2cyclictest7491-21ssh00:15:160
2662099129,2cyclictest6735-21head23:55:180
2662099129,2cyclictest30428-21tr20:25:140
2662099129,2cyclictest2751-21munin-run23:15:010
2662099129,2cyclictest23326-21tr22:10:150
2662099129,2cyclictest20689-21sort21:30:000
2662099129,2cyclictest1423-21wc19:25:000
2662099128,3cyclictest25189-21expr00:25:180
2662099126,2cyclictest452-21systemd-logind00:07:230
2662099125,1cyclictest0-21swapper/021:05:000
2662099122,9cyclictest2692-21kworker/u8:223:19:070
2662099121,6cyclictest6875-21munin-run00:15:000
26620991212,0cyclictest0-21swapper/022:50:360
2663099119,1cyclictest9504-21grep19:40:163
2663099119,1cyclictest29834-21ntpq23:30:203
2663099119,1cyclictest29531-21sed20:20:263
2663099119,1cyclictest13201-21uniq20:55:163
2663099118,3cyclictest10584-21ssh23:19:323
2663099118,2cyclictest7995-21sed19:35:273
2663099118,2cyclictest6146-21sed21:20:253
2663099118,2cyclictest32676-21grep20:30:163
2663099118,2cyclictest26483-21cat20:15:183
2663099118,2cyclictest22775-21cron20:10:003
2663099118,2cyclictest22055-21ssh00:23:403
2663099118,2cyclictest20988-21ssh21:30:103
2663099118,2cyclictest1410-21ssh23:13:593
2663099118,2cyclictest11374-21sh22:21:433
2663099115,5cyclictest26614-21cyclictest20:00:213
2663099110,10cyclictest6255-21ntpq20:40:203
2663099110,10cyclictest2986-21grep20:35:153
2662599119,1cyclictest18392-21tr21:05:212
2662599118,2cyclictest9119-21sh22:00:542
2662599118,2cyclictest577-21cut22:15:212
2662599118,2cyclictest4655-21grep19:30:162
2662599118,2cyclictest3453-21grep20:35:172
2662599118,2cyclictest29757-21rm23:10:492
2662599118,2cyclictest25222-21sh21:32:042
2662599118,2cyclictest2422-21ssh21:57:172
2662599118,2cyclictest22817-21tail21:50:172
2662599118,2cyclictest20638-21ssh22:08:162
2662599118,2cyclictest20404-21idleruntime-cro20:05:002
2662599118,2cyclictest19153-21tr23:05:152
2662599118,2cyclictest14770-21ssh00:38:342
2662599118,2cyclictest14519-21chrt20:58:402
2662599115,2cyclictest225-21systemd-journal20:15:242
2662599114,6cyclictest25833-21wc00:25:242
2662599114,3cyclictest225-21systemd-journal19:46:462
2662599110,1cyclictest0-21swapper/223:42:562
2662299119,1cyclictest0-21swapper/123:20:441
2662299118,2cyclictest863-21awk22:35:141
2662299118,2cyclictest8243-21ssh00:15:191
2662299118,2cyclictest32132-21expr22:15:141
2662299118,2cyclictest29630-21chrt20:21:151
2662299118,2cyclictest27261-21chrt20:16:351
2662299118,2cyclictest2644-21ssh23:33:551
2662299118,2cyclictest24558-21ssh22:10:231
2662299118,2cyclictest2178-21tr22:55:141
2662299118,2cyclictest21135-21ssh23:05:401
2662299118,2cyclictest18021-21tail21:05:191
2662299118,2cyclictest16153-21tr21:00:221
2662299118,2cyclictest13471-21cut20:55:171
2662299118,2cyclictest1203-21wc20:30:221
2662299118,2cyclictest11947-21sh22:40:591
2662299118,2cyclictest11121-21chrt19:45:091
2662299117,3cyclictest9003-21tr19:40:151
2662299110,1cyclictest141rcu_preempt23:55:011
2662299110,11cyclictest0-21swapper/100:26:121
2662099119,1cyclictest323-21aten2_r7power_p00:30:130
2662099118,2cyclictest8831-21sed22:20:180
2662099118,2cyclictest29046-21tail21:15:160
2662099118,2cyclictest26066-21grep20:15:140
2662099118,2cyclictest25093-21ssh23:08:450
2662099118,2cyclictest20498-21rm21:10:160
2662099118,2cyclictest18130-21ssh23:23:480
2662099118,2cyclictest17125-21grep19:55:210
2662099118,2cyclictest15456-21expr22:05:160
2662099118,2cyclictest12213-21ssh21:44:170
2662099118,2cyclictest10916-21tail20:50:160
2662099113,6cyclictest28368-21df_abs20:20:160
2662099112,3cyclictest0-21swapper/020:35:140
2663099109,1cyclictest0-21swapper/319:45:123
2663099108,2cyclictest7823-21expr00:15:163
2663099108,2cyclictest1555-21ntp_states00:30:203
2663099108,1cyclictest9691-21taskset20:47:373
2663099108,1cyclictest8611-21rm00:35:113
2663099108,1cyclictest6292-21ssh22:00:063
2663099108,1cyclictest5303-21ssh21:40:113
2663099108,1cyclictest30339-21taskset21:55:113
2663099108,1cyclictest29205-21basename19:15:113
2663099108,1cyclictest27957-21ssh22:50:283
2663099108,1cyclictest17390-21ssh22:45:023
2663099107,2cyclictest24770-21grep20:10:293
2663099107,2cyclictest15426-21sort00:00:183
2663099104,5cyclictest0-21swapper/322:15:443
2662599108,1cyclictest31516-21fgrep20:25:222
2662599108,1cyclictest11915-21chrt20:50:412
2662599108,1cyclictest11392sleep219:22:592
2662599107,2cyclictest8926-21egrep19:40:142
2662599107,2cyclictest2539-21wc19:25:202
2662599107,2cyclictest24914-21taskset20:12:012
2662599107,2cyclictest14255-21tail19:50:162
2662299107,2cyclictest8504-21grep20:45:161
2662299107,2cyclictest5623-21grep20:40:141
2662299107,2cyclictest27341-21grep19:10:151
2662299107,2cyclictest1580-21taskset19:25:131
2662099108,1cyclictest5112-21ntpq19:30:200
2662099108,1cyclictest32717-21egrep20:30:160
2662099108,1cyclictest1947-21expr19:25:140
2662099108,1cyclictest12725-21cat20:55:130
266309997,2cyclictest4329-21expr19:30:143
266309997,1cyclictest14917-21apache_accesses21:00:123
266309997,1cyclictest1431-21cut19:25:003
266259991,4cyclictest225-21systemd-journal21:02:012
266209997,1cyclictest29453-21cut19:15:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional