You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-23 - 02:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed Sep 23, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2134129481,9sleep00-21swapper/019:08:210
2120328665,6sleep10-21swapper/119:06:341
2115428564,16sleep20-21swapper/219:05:542
1933828471,9sleep30-21swapper/319:05:073
304422620,0sleep10-21swapper/100:13:001
310452570,0sleep00-21swapper/000:32:300
250392490,1sleep339-21ksoftirqd/323:11:343
2156099300,29cyclictest0-21swapper/122:55:471
21568992926,3cyclictest11427-21kworker/3:222:30:223
2155899241,4cyclictest0-21swapper/022:40:120
2155899220,22cyclictest0-21swapper/021:42:450
2155899220,20cyclictest0-21swapper/021:17:190
2156399212,5cyclictest0-21swapper/219:55:142
2156399210,21cyclictest0-21swapper/221:10:232
2156399200,4cyclictest0-21swapper/221:55:132
2156099201,0cyclictest0-21swapper/121:30:311
21558992016,3cyclictest4481-21date23:00:010
2155899200,19cyclictest0-21swapper/000:20:290
21568991918,1cyclictest0-21swapper/323:04:043
2156399190,19cyclictest0-21swapper/222:38:002
2156899184,3cyclictest141rcu_preempt22:40:533
2156899182,5cyclictest0-21swapper/319:10:203
21568991816,2cyclictest8258-21kworker/3:123:26:423
21568991814,3cyclictest11427-21kworker/3:221:22:323
21568991813,4cyclictest39-21ksoftirqd/322:12:063
2155899180,2cyclictest0-21swapper/021:13:400
2155899180,13cyclictest9734-21ssh22:05:020
2156399170,1cyclictest0-21swapper/220:00:212
2156399170,16cyclictest0-21swapper/223:26:532
2156099170,4cyclictest141rcu_preempt00:36:121
2156899164,9cyclictest23685-21/usr/sbin/munin23:50:133
21568991614,2cyclictest23142-21kworker/3:000:21:023
21568991614,2cyclictest11427-21kworker/3:221:45:123
21568991613,3cyclictest8258-21kworker/3:100:07:123
2156399164,10cyclictest0-21swapper/223:00:142
2156399162,3cyclictest141rcu_preempt23:53:502
2156399162,3cyclictest0-21swapper/222:25:422
21563991614,1cyclictest0-21swapper/223:18:122
21568991512,2cyclictest11427-21kworker/3:222:35:023
21568991510,3cyclictest2324-21wc19:35:213
2156899150,15cyclictest0-21swapper/321:34:003
21563991513,1cyclictest55850irq/129-enp2s0-21:20:582
21560991512,2cyclictest19193-21cut22:10:221
21560991511,3cyclictest221irq_work/100:15:181
21560991510,3cyclictest0-21swapper/123:51:181
2155899150,1cyclictest0-21swapper/023:31:580
2155899150,1cyclictest0-21swapper/021:39:380
2155899150,1cyclictest0-21swapper/021:30:580
2155899150,14cyclictest0-21swapper/000:25:390
21568991412,2cyclictest11427-21kworker/3:222:20:223
21568991411,2cyclictest25888-21cron00:30:003
21568991410,3cyclictest24070-21grep21:15:183
2156899140,13cyclictest2582-21grep22:00:213
2156399147,3cyclictest225-21systemd-journal20:56:112
21563991410,3cyclictest22757-21grep23:10:252
21563991410,3cyclictest19425-21tr20:10:212
2156399140,13cyclictest18357-21wc21:50:192
2156399140,12cyclictest5342-21grep19:40:272
2156399140,11cyclictest0-21swapper/222:22:472
2156099144,2cyclictest225-21systemd-journal19:19:381
21560991413,1cyclictest241rcuc/122:41:351
21560991413,1cyclictest0-21swapper/123:06:301
21560991411,2cyclictest27377-21sh22:35:011
21560991410,1cyclictest25-21ksoftirqd/123:03:421
2156099140,3cyclictest141rcu_preempt22:05:161
2156099140,3cyclictest0-21swapper/100:22:571
2156099140,0cyclictest0-21swapper/121:52:161
2155899143,7cyclictest0-21swapper/023:20:110
2155899141,7cyclictest5774-21munin-run19:45:000
2156899139,3cyclictest854-21sort21:40:153
2156899139,3cyclictest7893-21cat21:25:163
2156899139,3cyclictest18909-21grep20:10:163
21568991312,1cyclictest8258-21kworker/3:123:43:023
21568991311,2cyclictest8258-21kworker/3:123:21:423
21568991311,2cyclictest11427-21kworker/3:221:55:223
21568991310,2cyclictest7077-21wc20:55:003
21568991310,2cyclictest31045-21latency_hist19:30:003
21568991310,2cyclictest23435-21sed23:30:153
21568991310,2cyclictest17695-21ssh21:50:153
21568991310,2cyclictest16627-21grep21:10:303
21568991310,2cyclictest15403-21grep23:45:153
21568991310,2cyclictest13306-21sed22:45:203
2156899130,12cyclictest15548-21cron20:05:003
2156399139,3cyclictest22026-21wc20:15:252
2156399139,3cyclictest1819-21wc21:40:242
2156399138,4cyclictest7184-21tr23:40:132
21563991312,1cyclictest32-21ksoftirqd/200:25:462
21563991310,2cyclictest7069-21wc23:20:202
21563991310,2cyclictest5046-21ssh23:38:582
21563991310,2cyclictest2303-21grep22:00:192
21563991310,2cyclictest20798-21tr22:50:142
21563991310,2cyclictest13873-21wc22:45:262
21563991310,2cyclictest12025-21head22:05:262
21563991310,2cyclictest10070-21tail21:45:212
2156399130,1cyclictest141rcu_preempt00:07:372
2156399130,13cyclictest0-21swapper/200:36:412
2156399130,12cyclictest0-21swapper/223:08:252
2156099139,3cyclictest27456-21sh20:30:001
2156099139,3cyclictest26153-21grep21:35:261
2156099139,3cyclictest23130-21ssh00:09:051
2156099134,5cyclictest225-21systemd-journal21:25:001
2156099131,9cyclictest0-21swapper/120:35:121
21560991310,2cyclictest874-21cron22:00:001
21560991310,2cyclictest32751-21sort23:55:171
21560991310,2cyclictest29735-21grep22:35:261
21560991310,2cyclictest13013-21sed22:45:181
21560991310,2cyclictest1286-21ls20:40:211
21560991310,2cyclictest11327-21tr19:55:151
2156099130,13cyclictest0-21swapper/123:45:411
2156099130,0cyclictest0-21swapper/121:44:421
21558991310,2cyclictest32276-21grep23:55:150
21558991310,2cyclictest28744-21wc19:25:000
21558991310,2cyclictest26320-21sed21:55:160
21558991310,2cyclictest22077-21grep19:10:210
21558991310,2cyclictest21977-21grep23:10:200
21558991310,2cyclictest21589-21ssh22:50:190
21558991310,2cyclictest1094-21sed21:20:290
2155899130,7cyclictest9187-21awk00:20:000
2155899130,2cyclictest0-21swapper/000:09:180
2155899130,12cyclictest28125-21head22:15:220
2155899130,12cyclictest0-21swapper/019:50:150
2156899129,3cyclictest11427-21kworker/3:222:06:323
2156899129,2cyclictest5158-21chrt00:36:063
2156899129,2cyclictest4560-21ls20:50:003
2156899129,2cyclictest27288-21grep22:15:163
2156899129,2cyclictest2662-21chrt22:58:443
2156899129,2cyclictest21842-21wc20:15:223
2156899129,2cyclictest17922-21cron20:10:003
2156899128,4cyclictest24341-21apache200:12:163
2156899128,2cyclictest21312-21grep22:50:173
2156899126,1cyclictest32125-21ssh00:33:223
2156399129,2cyclictest4456-21grep23:00:002
2156399129,2cyclictest30553-21grep19:25:262
2156399129,2cyclictest23443-21cut23:30:152
2156399129,2cyclictest131792sleep221:05:232
2156399128,3cyclictest26301-21systemctl19:20:002
21563991210,1cyclictest9974-21tr00:20:142
2156399120,12cyclictest0-21swapper/221:28:322
2156399120,11cyclictest27754-21grep00:30:202
2156099129,2cyclictest6920-21cron20:55:001
2156099129,2cyclictest30074-21basename23:15:151
2156099129,2cyclictest29555-21tune2fs19:25:161
2156099129,2cyclictest2774-21cat19:35:271
2156099129,2cyclictest23417-21head23:30:151
2156099129,2cyclictest12753-21head21:05:191
2156099129,2cyclictest10412-21head21:00:171
2156099129,2cyclictest10412-21head21:00:171
2156099128,3cyclictest29250-21ssh00:31:061
2156099128,3cyclictest21323-21tail20:15:171
2156099128,3cyclictest15808-21cron00:05:001
2156099128,3cyclictest15734-21grep23:25:221
2156099127,4cyclictest783-21ssh23:35:381
2156099125,4cyclictest9973-21apache223:11:511
2156099123,8cyclictest11098-21ssh22:25:141
21560991210,2cyclictest14144-21diskmemload22:24:141
2156099120,2cyclictest0-21swapper/121:46:561
2156099120,11cyclictest0-21swapper/122:03:151
2155899129,2cyclictest7533-21ssh23:40:160
2155899129,2cyclictest31728-21cat23:15:260
2155899129,2cyclictest28712-21cat20:30:190
2155899129,2cyclictest23212-21tr20:20:150
2155899129,2cyclictest13288-21cat22:45:200
2155899129,2cyclictest10673-21cat21:00:200
2155899129,2cyclictest10673-21cat21:00:200
2155899128,2cyclictest10208-21ls21:45:230
21558991210,1cyclictest0-21swapper/022:20:580
2155899120,11cyclictest11778-21head19:55:170
2156899118,2cyclictest7239-21cut20:55:143
2156899118,2cyclictest4997-21tr19:40:233
2156899118,2cyclictest458-21expr20:40:153
2156899118,2cyclictest3564-21taskset00:15:393
2156899118,2cyclictest32650-21ssh23:55:163
2156899118,2cyclictest32518-21ssh23:15:543
2156899118,2cyclictest25188-21load19:15:193
2156899118,2cyclictest25002-21taskset20:24:023
2156899118,2cyclictest15037-21wc23:05:273
2156899118,2cyclictest1324-21ssh23:36:043
2156899118,2cyclictest12329-21ssh00:02:163
2156899118,2cyclictest10506-21taskset19:51:323
2156899117,3cyclictest13049-21grep22:25:273
2156899114,1cyclictest24264-21apache_processe21:35:123
2156399118,2cyclictest9399-21ssh00:00:232
2156399118,2cyclictest932-21cat20:40:182
2156399118,2cyclictest6260-21tr19:45:142
2156399118,2cyclictest6017-21sed20:50:232
2156399118,2cyclictest4220-21df22:40:132
2156399118,2cyclictest4085-21ssh00:16:032
2156399118,2cyclictest29504-21ssh22:16:042
2156399118,2cyclictest25139-21ssh21:35:172
2156399118,2cyclictest23626-21sort20:20:172
2156399118,2cyclictest23283-21ssh21:15:142
2156399118,2cyclictest18617-21ssh21:30:592
2156399118,2cyclictest15442-21grep23:45:152
2156399118,2cyclictest1447-21grep23:55:262
2156399116,1cyclictest32-21ksoftirqd/200:10:252
21563991110,1cyclictest0-21swapper/222:32:452
2156399110,4cyclictest225-21systemd-journal20:35:392
2156399110,3cyclictest0-21swapper/222:11:412
2156099119,1cyclictest7123-21tr23:40:131
2156099118,2cyclictest29439-21ssh22:16:011
2156099118,2cyclictest23755-21cut20:20:171
2156099118,2cyclictest23421-21ssh22:51:101
2156099118,2cyclictest20874-21sh00:26:081
2156099118,2cyclictest11370-21ssh23:23:081
2155899119,1cyclictest32370-21head23:35:220
2155899119,1cyclictest21342-21sed20:15:180
2155899119,1cyclictest18024-21grep21:50:170
2155899118,2cyclictest8091-21ssh23:01:260
2155899118,2cyclictest7332-21ssh21:25:060
2155899118,2cyclictest32196-21cron20:40:000
2155899118,2cyclictest29952-21ssh22:35:340
2155899118,2cyclictest2743-21ntpq22:00:220
2155899118,2cyclictest2723-21grep20:45:180
2155899118,2cyclictest25714-21ls19:15:260
2155899118,2cyclictest25213-21bc00:10:140
2155899118,2cyclictest24413-21grep23:50:170
2155899118,2cyclictest2005-21head19:35:180
2155899118,2cyclictest18090-21sh22:10:010
2155899118,2cyclictest17760-21ssh23:26:350
2155899118,2cyclictest1765-21sed20:40:260
2155899118,2cyclictest16824-21wc23:45:230
2155899118,2cyclictest15100-21sed20:00:260
2155899118,2cyclictest13040-21grep22:25:270
2155899118,2cyclictest12334-21cstates21:05:170
2155899111,4cyclictest0-21swapper/020:50:160
2155899110,10cyclictest9757-21ssh00:39:420
2155899110,10cyclictest7800-21ssh00:00:070
2156899108,1cyclictest32684-21grep19:30:223
2156899108,1cyclictest30311-21ls20:35:133
2156899108,1cyclictest13608-21cat21:05:283
2156899107,2cyclictest6100-21wget19:45:133
2156899107,2cyclictest28171-21sed19:20:263
2156899107,2cyclictest25445-21basename20:25:143
2156899107,2cyclictest11002-21sendmail_mailtr21:00:223
2156899107,2cyclictest11002-21sendmail_mailtr21:00:223
2156899100,8cyclictest0-21swapper/320:30:143
2156399108,1cyclictest9983-21gzip19:50:232
2156399108,1cyclictest4523-21chrt20:49:442
2156399108,1cyclictest27860-21cut20:30:152
2156399108,1cyclictest27188-21sshd20:26:382
2156399108,1cyclictest1224-21chrt19:35:072
2156399107,2cyclictest31563-21tr19:30:142
2156399107,2cyclictest27320-21sed19:20:172
2156399103,6cyclictest22491-21fschecks_time19:10:242
2156099108,1cyclictest9821-21cut19:50:231
2156099108,1cyclictest6353-21expr19:45:141
2156099108,1cyclictest31276-21taskset19:30:091
2156099108,1cyclictest297712sleep120:34:211
2156099108,1cyclictest22927-21ntpq19:10:271
2156099108,1cyclictest15853-21cut20:05:141
2156099108,1cyclictest10162-21ssh21:25:541
2156099107,2cyclictest8276-21cat20:55:201
2156099107,2cyclictest4848-21pluginstate19:40:221
2156099107,2cyclictest3583-21tr20:45:241
2156099107,2cyclictest26565-21chrt19:20:121
2156099107,2cyclictest25574-21ssh21:15:411
2156099107,2cyclictest15255-21expr21:10:191
2156099107,2cyclictest14160-21expr20:00:171
2155899108,1cyclictest25455-21wget20:25:140
2155899108,1cyclictest19858-21expr22:30:160
2155899108,1cyclictest16269-21expr20:05:150
2155899108,1cyclictest12974-21seq23:05:020
2155899108,1cyclictest1123-21cat19:35:010
2155899107,2cyclictest6295-21grep19:45:140
2155899107,2cyclictest29763-21basename19:25:180
2155899107,2cyclictest19071-21if_enp2s020:10:180
215689996,2cyclictest11397-21expr19:55:153
215639997,1cyclictest177512sleep220:07:492
215639995,1cyclictest9553-21taskset21:00:092
215639995,1cyclictest9553-21taskset21:00:092
215609997,1cyclictest19880-21grep20:10:261
215589986,1cyclictest7190-21apache_accesses20:55:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional