You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-12 - 11:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sun Jul 12, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2398921040,0sleep00-21swapper/022:43:190
311128968,16sleep10-21swapper/119:09:411
311128968,16sleep10-21swapper/119:09:401
308528974,11sleep20-21swapper/219:09:222
308528974,11sleep20-21swapper/219:09:212
285228976,9sleep30-21swapper/319:06:213
285228976,9sleep30-21swapper/319:06:213
242828978,6sleep00-21swapper/019:05:260
242828978,6sleep00-21swapper/019:05:250
301772580,0sleep10-21swapper/121:49:241
322799314,25cyclictest12481-21/usr/sbin/munin23:15:181
322799270,3cyclictest7419-21hwlatdetect23:50:191
322799251,5cyclictest6998-21munin-run21:55:001
322799250,24cyclictest28488-21diskmemload23:41:141
322799245,7cyclictest28488-21diskmemload22:25:251
323499231,1cyclictest0-21swapper/323:23:483
322799236,1cyclictest28488-21diskmemload21:24:531
322799230,1cyclictest0-21swapper/122:00:151
322799224,3cyclictest0-21swapper/100:35:561
3227992219,2cyclictest9240-21apt-get20:30:141
322799221,2cyclictest225-21systemd-journal21:10:001
322799221,1cyclictest0-21swapper/120:00:131
3234992120,1cyclictest0-21swapper/322:20:543
323099210,19cyclictest0-21swapper/200:17:452
3224992011,7cyclictest13-21ksoftirqd/021:47:330
322499200,1cyclictest0-21swapper/021:56:140
323499190,2cyclictest0-21swapper/323:10:193
323099190,19cyclictest0-21swapper/222:14:472
322799196,13cyclictest0-21swapper/122:17:301
322799192,3cyclictest771-21df_inode22:10:131
322799190,5cyclictest28488-21diskmemload00:34:331
322799190,16cyclictest0-21swapper/122:45:161
322799190,16cyclictest0-21swapper/122:45:151
322799190,0cyclictest0-21swapper/122:44:531
322499190,3cyclictest0-21swapper/000:19:320
323499180,14cyclictest0-21swapper/322:05:203
323099180,18cyclictest0-21swapper/223:29:322
323099180,15cyclictest0-21swapper/222:15:022
322799186,11cyclictest15148-21apt19:35:001
322799183,1cyclictest28488-21diskmemload21:14:371
3227991814,4cyclictest0-21swapper/100:03:291
3227991813,4cyclictest25-21ksoftirqd/123:23:121
322799180,3cyclictest0-21swapper/123:28:161
322799180,18cyclictest0-21swapper/122:50:571
322799180,17cyclictest32219-21grep21:30:261
322499180,3cyclictest0-21swapper/023:35:250
322499180,18cyclictest0-21swapper/023:58:580
322499180,13cyclictest0-21swapper/023:00:110
59662170,1chrt0-21swapper/021:34:590
323099177,5cyclictest225-21systemd-journal21:00:002
3230991715,2cyclictest10386-21kworker/u8:100:08:192
3227991715,1cyclictest0-21swapper/123:30:021
3227991713,4cyclictest0-21swapper/122:09:421
322799171,15cyclictest7487-21sort20:25:171
322799170,15cyclictest0-21swapper/121:27:071
322799170,0cyclictest0-21swapper/100:13:371
3224991717,0cyclictest0-21swapper/022:04:340
3224991714,2cyclictest0-21swapper/022:28:380
322499170,17cyclictest0-21swapper/021:42:210
323499162,0cyclictest0-21swapper/323:28:403
323499161,14cyclictest0-21swapper/319:50:133
3230991615,1cyclictest0-21swapper/222:07:532
322799169,7cyclictest0-21swapper/123:09:431
322799169,6cyclictest22497-21taskset23:01:041
322799168,7cyclictest18240-21chrt22:20:211
322799168,2cyclictest23186-21chrt20:56:491
322799168,2cyclictest14188-21/usr/sbin/munin21:40:001
322799167,8cyclictest6423-21taskset20:22:561
322799164,4cyclictest21880-21sh21:44:271
322799163,11cyclictest28488-21diskmemload21:15:461
322499166,5cyclictest27063-21perf20:00:000
3224991612,1cyclictest0-21swapper/000:14:500
322499160,1cyclictest0-21swapper/022:09:410
322499160,0cyclictest0-21swapper/022:16:510
3234991513,1cyclictest9652-21ssh21:55:333
3230991511,2cyclictest14711-21wc21:20:192
322799158,6cyclictest5559-21sh19:15:011
322799158,6cyclictest22758-21ls23:59:591
322799158,6cyclictest20825-21ssh00:17:251
322799158,2cyclictest5336-21ssh00:08:241
322799158,2cyclictest13883-21ssh21:58:521
322799158,1cyclictest24136-21cat19:50:261
3227991513,1cyclictest0-21swapper/122:55:141
322799150,4cyclictest0-21swapper/122:38:081
322799150,3cyclictest0-21swapper/100:28:231
322799150,1cyclictest0-21swapper/119:45:131
322499152,6cyclictest5572-21munin-run19:15:010
3224991512,2cyclictest161rcuc/022:50:220
323499148,5cyclictest39-21ksoftirqd/322:12:013
3234991412,2cyclictest39-21ksoftirqd/323:01:193
3234991411,2cyclictest32238-21tail21:30:273
3234991411,2cyclictest16371-21latency_hist22:20:003
3234991410,3cyclictest14935-21grep20:40:173
3234991410,2cyclictest6372-21grep21:35:133
3230991411,2cyclictest5926-21apache221:41:072
3230991410,3cyclictest21526-21cron23:40:002
323099140,6cyclictest0-21swapper/220:49:072
323099140,3cyclictest0-21swapper/200:13:242
322799148,5cyclictest2639-21tune2fs20:15:151
322799148,5cyclictest12689-21chrt19:29:071
322799148,2cyclictest27613-21ssh00:21:081
322799148,2cyclictest18376-21ssh23:37:321
322799148,1cyclictest7140-21ssh22:33:151
322799147,2cyclictest861-21ssh23:46:051
322799147,2cyclictest4725-21sort23:10:161
322799147,2cyclictest26849-21taskset19:57:451
322799147,2cyclictest18040-21cat20:45:251
322799147,2cyclictest1327-21grep20:10:241
322799147,1cyclictest7337-21grep19:15:271
322799141,3cyclictest225-21systemd-journal21:01:331
322499142,1cyclictest141rcu_preempt23:23:530
3224991410,3cyclictest7082-21cat23:30:240
3224991410,3cyclictest18532-21tail19:40:160
322499140,7cyclictest6617-21awk20:25:000
322499140,1cyclictest0-21swapper/000:36:190
323499138,3cyclictest18294-21df19:40:153
323499134,6cyclictest28992-21apt-get21:10:143
3234991312,1cyclictest0-21swapper/321:18:033
3234991310,2cyclictest27455-21grep22:45:193
3234991310,2cyclictest27455-21grep22:45:183
3234991310,2cyclictest18713-21munin-run20:49:593
3234991310,2cyclictest17510-21cron19:40:013
3234991310,2cyclictest12804-21wc22:55:233
3234991310,2cyclictest1200-21grep00:05:253
323499130,12cyclictest23464-21awk21:00:013
323099139,3cyclictest5846-21grep21:15:192
323099132,9cyclictest10408-21latency_hist19:25:012
3230991312,1cyclictest0-21swapper/219:55:232
3230991310,2cyclictest9484-21ssh00:30:142
3230991310,2cyclictest8531-21grep20:25:272
3230991310,2cyclictest8171-21wc23:50:262
3230991310,2cyclictest5755-21wc19:15:012
3230991310,2cyclictest28338-21wc22:45:282
3230991310,2cyclictest28338-21wc22:45:272
3230991310,2cyclictest22068-21basename21:25:132
3230991310,2cyclictest20404-21readlink23:20:002
3230991310,2cyclictest19520-21grep22:40:202
323099130,1cyclictest0-21swapper/223:03:482
323099130,12cyclictest0-21swapper/222:25:252
322799138,4cyclictest152382sleep120:40:201
322799136,5cyclictest15416-21apt-get19:35:141
322799131,5cyclictest10389-21cron19:25:011
322499139,3cyclictest3287-21grep22:30:250
322499139,3cyclictest30712-21wc21:10:270
322499132,7cyclictest141rcu_preempt23:10:010
3224991310,2cyclictest426-21tr22:10:120
3224991310,2cyclictest20467-21grep19:45:140
3224991310,2cyclictest20415-21tr23:20:010
322499130,12cyclictest17130-21ssh22:39:320
323499129,2cyclictest9206-21cut00:30:133
323499129,2cyclictest8674-21grep00:10:173
323499129,2cyclictest6569-21sort19:15:173
323499129,2cyclictest5776-21grep20:20:233
323499129,2cyclictest2823-21grep22:30:213
323499129,2cyclictest24949-21grep21:00:223
323499129,2cyclictest24594-21tr00:00:213
323499129,2cyclictest22745-21sort23:40:153
323499129,2cyclictest20893-21head19:45:173
323499129,2cyclictest20072-21uname20:50:213
323499129,2cyclictest174772sleep322:00:233
323499129,2cyclictest15991-21cat23:55:203
323499129,2cyclictest14444-21head21:20:173
323499129,2cyclictest10659-21taskset22:35:163
323499128,3cyclictest39-21ksoftirqd/300:29:073
323499123,7cyclictest24392-21latency_hist00:20:003
323499121,10cyclictest0-21swapper/300:38:173
323499120,1cyclictest0-21swapper/321:41:113
323099129,2cyclictest9945-21basename22:35:142
323099129,2cyclictest6071-21grep23:10:272
323099129,2cyclictest30482-21head20:05:192
323099129,2cyclictest24451-21seq23:21:392
323099129,2cyclictest22922-21head23:40:162
323099129,2cyclictest19747-21cut20:50:182
323099129,2cyclictest18311-21sort00:35:202
323099128,3cyclictest9539-21grep21:55:302
323099126,6cyclictest5926-21apache221:49:262
323099125,5cyclictest15147-21latency_hist19:35:012
323099123,9cyclictest32-21ksoftirqd/223:56:172
323099120,9cyclictest1142-21runrttasks21:31:072
323099120,1cyclictest0-21swapper/221:52:302
323099120,12cyclictest0-21swapper/200:22:102
322799129,2cyclictest29942-21grep20:05:151
322799129,2cyclictest19732-21sed20:50:181
322799122,2cyclictest225-21systemd-journal20:37:351
322499129,2cyclictest4104-21grep23:10:130
322499129,2cyclictest31577-21ssh00:05:090
322499129,2cyclictest30784-21grep20:05:200
322499129,2cyclictest27015-21sort21:05:180
322499129,2cyclictest25885-21apt-get21:05:010
322499129,2cyclictest25359-21ssh00:20:150
322499129,2cyclictest17542-21awk19:40:010
322499129,2cyclictest14468-21sed21:20:170
322499122,8cyclictest13036-21apache_processe19:30:130
322499120,0cyclictest0-21swapper/022:48:310
322499120,0cyclictest0-21swapper/022:48:300
323499118,2cyclictest7214-21grep23:30:263
323499118,2cyclictest7196-21expr20:25:153
323499118,2cyclictest5405-21taskset19:13:083
323499118,2cyclictest32660-21fschecks_count20:10:143
323499118,2cyclictest30864-21ssh23:06:023
323499118,2cyclictest30863-21expr23:45:153
323499118,2cyclictest28226-21taskset21:07:013
323499118,2cyclictest26825-21taskset19:57:283
323499118,2cyclictest25271-21ssh22:25:033
323499118,2cyclictest25256-21ssh00:20:163
323499118,2cyclictest24551-21ssh21:25:523
323499118,2cyclictest2436-21ssh22:50:053
323499118,2cyclictest2109-21ssh21:51:143
323499118,2cyclictest19370-21ssh23:38:183
323499118,2cyclictest18264-21ssh23:18:223
323499118,2cyclictest14924-21chrt19:32:183
323499118,2cyclictest10769-21grep19:25:153
323499118,2cyclictest10536-21ssh23:52:003
323499118,2cyclictest10014-21head20:30:173
323499111,10cyclictest18923-21diskstats22:40:163
3234991110,1cyclictest0-21swapper/321:45:153
323099119,2cyclictest21674-21kworker/u8:223:05:092
323099119,1cyclictest0-21swapper/219:35:132
323099118,2cyclictest956-21fgrep20:10:202
323099118,2cyclictest9365-21cut20:30:142
323099118,2cyclictest5784-21ssh00:27:492
323099118,2cyclictest5568-21ssh22:32:052
323099118,2cyclictest5198-21ssh22:50:532
323099118,2cyclictest5058-21ssh23:30:052
323099118,2cyclictest4919-21grep20:20:162
323099118,2cyclictest4103-21ssh23:48:362
323099118,2cyclictest29369-21expr21:10:152
323099118,2cyclictest28245-21grep20:00:192
323099118,2cyclictest23589-21tr00:00:152
323099118,2cyclictest19538-21ssh22:01:432
323099118,2cyclictest17777-21head22:20:172
323099118,2cyclictest15588-21sh22:57:132
323099118,2cyclictest12608-21chrt19:28:042
323099118,2cyclictest12179-21chrt20:35:152
323099118,2cyclictest10607-21ssh21:37:132
323099112,5cyclictest49-21kcompactd019:18:162
323099111,5cyclictest225-21systemd-journal21:05:012
323099110,10cyclictest24323-21chrt19:50:472
322799118,2cyclictest17766-21chrt19:40:041
322499119,1cyclictest30478-21dpkg23:45:120
322499118,2cyclictest9956-21sed19:20:280
322499118,2cyclictest5496-21sh21:53:540
322499118,2cyclictest4739-21chrt21:15:050
322499118,2cyclictest29549-21grep23:25:150
322499118,2cyclictest29263-21chrt20:03:000
322499118,2cyclictest2675-21sed00:25:270
322499118,2cyclictest22822-21grep19:50:150
322499118,2cyclictest21789-21ssh23:40:010
322499118,2cyclictest18229-21ssh22:20:210
322499118,2cyclictest14361-21egrep20:40:130
322499118,2cyclictest11813-21ssh00:30:580
322499118,2cyclictest10894-21grep19:25:170
322499118,2cyclictest10007-21head20:30:170
322499117,3cyclictest7433-21cat19:15:270
322499117,3cyclictest22863-21wc21:25:160
322499117,3cyclictest12148-21grep22:55:190
322499115,2cyclictest225-21systemd-journal20:19:590
322499114,5cyclictest31175-21sh00:05:000
322499112,8cyclictest23463-21sleep21:00:010
323499108,1cyclictest3690-21awk20:15:263
323499100,9cyclictest27389-21wget20:00:143
323099108,1cyclictest26266-21find21:05:002
323099108,1cyclictest18123-21expr19:40:142
323099107,2cyclictest2750-21head20:15:152
323099107,2cyclictest21945-21taskset19:45:452
322499108,1cyclictest6740-21cat21:35:140
322499108,1cyclictest6464-21ssh23:50:120
322499108,1cyclictest32506-21ls20:10:130
322499107,2cyclictest19392-21cat20:50:160
322499107,2cyclictest17286-21cut20:45:170
322499107,2cyclictest11583-21cat20:35:120
32349997,1cyclictest8493-21grep19:20:163
32349997,1cyclictest31002-21gzip20:05:243
32349997,1cyclictest11991-21cat20:35:143
32309996,2cyclictest16117-21chrt20:42:292
32249997,1cyclictest6820-21chrt20:25:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional