You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-21 - 17:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sun Jun 21, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
570629380,9sleep00-21swapper/007:09:510
543728666,9sleep20-21swapper/207:06:242
559428269,9sleep30-21swapper/307:08:263
563728164,13sleep10-21swapper/107:09:001
5808992924,3cyclictest27952-21cron10:25:002
580899280,24cyclictest0-21swapper/209:46:002
580899260,25cyclictest0-21swapper/212:35:012
580499250,25cyclictest0-21swapper/110:23:221
580899240,23cyclictest0-21swapper/212:10:542
580899240,23cyclictest0-21swapper/211:19:502
580499240,22cyclictest0-21swapper/112:25:271
293422230,0sleep10-21swapper/110:05:291
5808992221,0cyclictest0-21swapper/208:05:142
5808992219,2cyclictest1853-21munin-run12:04:592
5808992119,1cyclictest0-21swapper/211:00:132
580499212,12cyclictest0-21swapper/111:11:021
580499210,1cyclictest0-21swapper/110:44:191
580099215,11cyclictest0-21swapper/011:05:130
581099200,0cyclictest0-21swapper/310:01:513
580499200,20cyclictest0-21swapper/112:07:151
5800992019,1cyclictest13-21ksoftirqd/011:20:040
580899190,19cyclictest0-21swapper/211:50:522
580899190,18cyclictest0-21swapper/211:09:332
5804991916,2cyclictest0-21swapper/112:37:361
580499190,1cyclictest0-21swapper/110:31:391
580499190,18cyclictest0-21swapper/110:45:281
580499190,18cyclictest0-21swapper/109:50:571
580499190,0cyclictest0-21swapper/111:56:091
581099180,17cyclictest3076-21ssh10:10:053
5808991817,1cyclictest0-21swapper/210:30:552
581099176,1cyclictest141rcu_preempt12:28:313
581099170,16cyclictest0-21swapper/311:22:103
580899172,0cyclictest3619-21strings12:05:212
580899170,17cyclictest0-21swapper/210:47:212
580499178,5cyclictest38150irq/131-snd_hda09:49:081
5804991717,0cyclictest0-21swapper/111:08:551
5804991713,3cyclictest0-21swapper/111:44:371
580499171,15cyclictest31021-21diskmemload10:52:521
580499170,1cyclictest0-21swapper/110:13:531
5800991716,1cyclictest0-21swapper/009:21:440
5810991615,1cyclictest0-21swapper/311:01:323
580899160,1cyclictest0-21swapper/210:40:222
580899160,16cyclictest0-21swapper/209:55:572
5804991615,1cyclictest0-21swapper/109:55:121
5804991613,2cyclictest0-21swapper/111:54:401
5804991612,3cyclictest27667-21cron07:55:001
580499160,16cyclictest0-21swapper/111:22:141
580499160,16cyclictest0-21swapper/110:39:261
580499160,16cyclictest0-21swapper/109:36:531
580099160,15cyclictest0-21swapper/012:07:180
580099160,15cyclictest0-21swapper/010:19:580
581099153,11cyclictest26848-21grep09:45:183
5810991512,2cyclictest11404-21rm12:10:143
5810991511,2cyclictest0-21swapper/309:25:013
581099150,14cyclictest0-21swapper/311:50:473
580899150,15cyclictest0-21swapper/212:24:372
580899150,15cyclictest0-21swapper/211:47:522
580899150,14cyclictest0-21swapper/210:15:442
580499158,6cyclictest38150irq/131-snd_hda09:20:201
580499151,1cyclictest31021-21diskmemload11:36:291
580499150,1cyclictest31021-21diskmemload10:26:071
580099155,5cyclictest31041-21perf09:10:000
580099154,9cyclictest9573-21awk08:25:000
580099150,2cyclictest0-21swapper/012:13:070
580099150,15cyclictest0-21swapper/010:06:520
580099150,0cyclictest0-21swapper/009:12:460
5810991411,2cyclictest12848-21grep08:30:173
581099140,10cyclictest0-21swapper/308:00:143
581099140,0cyclictest0-21swapper/307:50:153
580899142,5cyclictest0-21swapper/210:07:322
5808991413,1cyclictest0-21swapper/212:25:422
5808991413,1cyclictest0-21swapper/209:11:262
5808991411,2cyclictest17621-21sh09:20:242
580899140,14cyclictest0-21swapper/209:38:042
580899140,14cyclictest0-21swapper/209:34:222
580899140,14cyclictest0-21swapper/209:17:182
580899140,14cyclictest0-21swapper/209:17:172
580499149,3cyclictest25-21ksoftirqd/112:30:201
580499142,11cyclictest31021-21diskmemload11:32:341
580499141,13cyclictest0-21swapper/112:21:031
5804991411,2cyclictest19740-21head10:00:171
5804991410,3cyclictest6071-21grep07:10:131
580499140,1cyclictest0-21swapper/109:40:421
580499140,14cyclictest0-21swapper/107:15:171
580499140,13cyclictest27624-21tail09:00:201
5800991411,2cyclictest29544-21grep07:55:250
5800991410,3cyclictest7286-21cat08:20:000
580099140,8cyclictest27978-21idleruntime-cro10:25:000
581099132,1cyclictest0-21swapper/312:20:123
5810991312,1cyclictest0-21swapper/311:39:053
5810991310,2cyclictest833-21sh09:10:243
5810991310,2cyclictest29213-21grep10:05:293
5810991310,2cyclictest29140-21cron10:44:593
5810991310,2cyclictest24120-21uname07:45:163
5810991310,2cyclictest22367-21grep12:35:263
5810991310,2cyclictest10230-21cat11:30:243
581099130,12cyclictest15452-21cron09:20:013
581099130,12cyclictest15452-21cron09:20:003
580899139,3cyclictest7152-21cron08:20:002
580899139,3cyclictest31941-21sed08:00:272
580899139,3cyclictest18559-21wc08:40:262
580899139,3cyclictest14489-21wc07:25:192
580899139,3cyclictest13563-21chrt10:54:522
580899139,3cyclictest13486-21head08:30:232
580899139,2cyclictest9739-21cat08:24:592
580899139,2cyclictest10046-21sed07:15:282
5808991313,0cyclictest0-21swapper/210:11:502
5808991310,2cyclictest26168-21wc09:25:252
580899130,3cyclictest0-21swapper/211:20:302
580899130,1cyclictest0-21swapper/211:40:212
580499139,3cyclictest31010-21cron09:10:001
580499137,2cyclictest225-21systemd-journal11:05:011
5804991310,2cyclictest31789-21grep09:10:131
580499130,1cyclictest0-21swapper/112:01:491
580499130,0cyclictest0-21swapper/108:06:141
580099133,3cyclictest21496-21idleruntime-cro08:50:000
5800991310,2cyclictest32290-21ssh10:45:450
5800991310,2cyclictest3111-21grep11:45:250
5800991310,2cyclictest20747-21grep12:15:210
5800991310,2cyclictest20747-21grep12:15:200
5800991310,2cyclictest19197-21grep07:35:160
5800991310,2cyclictest14551-21grep10:55:140
5800991310,2cyclictest10499-21grep11:50:160
581099129,2cyclictest8553-21cut08:20:203
581099129,2cyclictest32333-21ntpq11:05:213
581099129,2cyclictest26764-21tr12:00:133
581099129,2cyclictest20944-21head10:20:173
581099129,2cyclictest18861-21grep11:55:153
581099129,2cyclictest14007-21grep12:30:273
581099129,2cyclictest11585-21grep08:25:263
581099128,3cyclictest26462-21grep11:40:183
581099128,3cyclictest19047-21ssh09:40:243
581099128,3cyclictest16297-21tr07:30:153
581099126,2cyclictest10556-21idleruntime-cro07:20:013
581099124,4cyclictest225-21systemd-journal07:45:013
580899129,2cyclictest2644-21df09:50:162
580899129,2cyclictest26098-21wc12:00:012
580899129,2cyclictest25354-21grep08:55:212
580899129,2cyclictest23376-21grep08:50:262
580899129,2cyclictest19308-21sed09:40:252
580899129,2cyclictest18832-21head11:35:272
580899129,2cyclictest18698-21tr07:35:142
580899129,2cyclictest10365-21grep08:25:152
580899128,2cyclictest27972-21grep07:55:122
580899126,2cyclictest225-21systemd-journal10:38:542
580899120,1cyclictest0-21swapper/210:26:162
580899120,12cyclictest0-21swapper/212:17:142
580899120,12cyclictest0-21swapper/212:17:132
580499129,2cyclictest24798-21sed07:45:241
580499129,2cyclictest22359-21wc07:40:251
580499129,2cyclictest21490-21cron08:50:001
580499129,2cyclictest18491-21grep07:35:131
580499129,2cyclictest16259-21head11:15:171
580499128,2cyclictest8253-21basename09:35:001
5804991210,1cyclictest15453-21cron09:20:011
5804991210,1cyclictest15453-21cron09:20:001
5804991210,1cyclictest12977-21sh07:25:011
580499120,11cyclictest13634-21grep10:15:261
580099129,2cyclictest9646-21cut09:35:150
580099129,2cyclictest9524-21cut07:15:210
580099129,2cyclictest8418-21sh11:10:210
580099129,2cyclictest28485-21wc12:40:000
580099129,2cyclictest27891-21head12:00:210
580099129,2cyclictest26116-21grep07:50:160
580099129,2cyclictest21587-21tr10:40:160
580099129,2cyclictest12355-21ssh12:30:160
580099128,2cyclictest15161-21grep07:25:240
580099124,7cyclictest22611-21/usr/sbin/munin11:00:260
5800991212,0cyclictest0-21swapper/011:32:440
5800991210,1cyclictest21777-21ssh11:37:420
580099120,11cyclictest5433-21cat12:25:250
581099119,1cyclictest25584-21needreboot09:25:183
581099119,1cyclictest16315-21grep11:15:173
581099118,2cyclictest8974-21rm10:32:253
581099118,2cyclictest7461-21expr11:10:153
581099118,2cyclictest7180-21basename08:20:003
581099118,2cyclictest6977-21ssh09:34:033
581099118,2cyclictest4769-21sh08:14:593
581099118,2cyclictest3564-21ssh11:45:423
581099118,2cyclictest32100-21ssh11:25:053
581099118,2cyclictest29999-21tail09:05:213
581099118,2cyclictest28248-21ssh10:25:043
581099118,2cyclictest23979-21cat08:55:003
581099118,2cyclictest20781-21wc12:15:213
581099118,2cyclictest20781-21wc12:15:213
581099118,2cyclictest15845-21ssh10:36:113
581099118,2cyclictest12904-21ssh10:54:193
581099114,3cyclictest3772-21grep12:05:233
581099110,1cyclictest0-21swapper/309:52:143
580899118,2cyclictest27475-21taskset07:52:362
580899118,2cyclictest20811-21sed10:00:272
580899118,2cyclictest20144-21sed08:45:152
580899118,2cyclictest16592-21df_abs07:30:162
580899118,2cyclictest14263-21ssh10:55:132
580899118,2cyclictest12001-21ssh11:12:432
580899117,2cyclictest32414-21tr11:25:132
580899117,2cyclictest24638-21wc07:45:242
580899113,4cyclictest225-21systemd-journal12:31:232
580499118,2cyclictest4373-21ssh11:46:191
580499118,2cyclictest30613-21tr08:00:141
580499118,2cyclictest29920-21chrt07:58:321
580499118,2cyclictest23863-21cron08:55:001
580499118,2cyclictest20510-21ssh10:58:441
580499118,2cyclictest18635-21sed08:40:261
580499118,2cyclictest12215-21chrt08:30:091
580499118,2cyclictest12016-21rm12:10:191
580499118,2cyclictest10525-21grep08:25:171
580499117,3cyclictest5757-21head08:15:181
580499117,3cyclictest20825-21cut12:15:211
580499117,3cyclictest20825-21cut12:15:201
580499111,6cyclictest225-21systemd-journal08:55:011
580499110,10cyclictest9199-21chrt08:20:361
580099119,1cyclictest14858-21grep08:35:150
580099118,2cyclictest986-21grep08:05:180
580099118,2cyclictest865-21expr09:30:140
580099118,2cyclictest5776-21ssh10:50:100
580099118,2cyclictest5626-21rm09:51:410
580099118,2cyclictest5158-21grep10:30:160
580099118,2cyclictest32474-21taskset12:22:350
580099118,2cyclictest31773-21ssh11:43:560
580099118,2cyclictest31040-21grep08:00:170
580099118,2cyclictest2912-21ssh11:26:170
580099118,2cyclictest28732-21grep10:25:130
580099118,2cyclictest26371-21tr09:45:160
580099118,2cyclictest24501-21wget09:25:140
580099118,2cyclictest24201-21ssh09:44:090
580099118,2cyclictest23136-21grep08:50:210
580099118,2cyclictest18497-21cat08:40:250
580099118,2cyclictest16305-21cut07:30:150
580099118,2cyclictest15775-21ssh10:36:080
580099118,2cyclictest15495-21tr09:20:010
580099118,2cyclictest15495-21tr09:20:010
580099118,2cyclictest13760-21cat08:30:270
580099118,2cyclictest11452-21seq07:20:160
580099117,2cyclictest20016-21cat10:00:190
580099114,2cyclictest11996-21munin-run08:29:590
5800991111,0cyclictest0-21swapper/011:17:200
580099110,10cyclictest18003-21/usr/sbin/munin11:55:200
581099108,1cyclictest30203-21grep10:45:163
581099108,1cyclictest28429-21grep07:55:143
581099108,1cyclictest27228-21grep09:00:183
581099108,1cyclictest2323-21sshd08:09:133
581099108,1cyclictest19599-21cat08:45:133
581099108,1cyclictest18737-21expr07:35:153
581099108,1cyclictest15421-21smartctl08:35:193
581099108,1cyclictest15080-21grep07:25:253
581099108,1cyclictest13028-21ntp_states10:15:213
581099108,1cyclictest10387-21sh09:55:103
581099107,2cyclictest17808-21chrt08:40:173
581099107,2cyclictest11630-21head07:20:163
581099107,2cyclictest10674-21cat09:35:243
581099100,9cyclictest0-21swapper/310:55:143
580899108,1cyclictest79472sleep207:12:362
580899108,1cyclictest3852-21taskset08:10:202
580899108,1cyclictest29034-21grep09:05:152
580899108,1cyclictest20861-21cut07:40:132
580899107,2cyclictest9013-21ssh11:30:142
580899107,2cyclictest26733-21sort09:00:162
580899100,9cyclictest166342sleep208:38:552
580499107,2cyclictest32242-21sh11:25:121
580499107,2cyclictest24647-21cut09:25:141
580499107,2cyclictest16725-21cut07:30:171
580499107,2cyclictest16264-21tail08:35:271
580499107,2cyclictest14558-21grep07:25:191
580499100,4cyclictest4767-21cron08:14:591
580099108,1cyclictest44272sleep008:10:410
580099108,1cyclictest26553-21cut09:00:140
580099108,1cyclictest23266-21taskset07:45:130
580099108,1cyclictest11118-21ssh09:55:130
580099107,2cyclictest6685-21grep07:10:170
580099107,2cyclictest2969-21ssh10:10:010
580099107,2cyclictest22993-21wc07:45:010
580099100,9cyclictest25906-21taskset08:55:460
58109998,1cyclictest26226-21taskset08:59:533
58109997,1cyclictest6248-21seq07:10:143
58089996,2cyclictest11605-21head07:20:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional