You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-18 - 12:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Mar 18, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
344729471,18sleep20-21swapper/219:06:402
362428676,6sleep00-21swapper/019:08:580
346128164,13sleep10-21swapper/119:06:501
358728067,9sleep30-21swapper/319:08:303
263202620,0sleep10-21swapper/122:45:001
379199270,27cyclictest0-21swapper/122:43:321
379699250,3cyclictest0-21swapper/321:51:333
379699250,1cyclictest0-21swapper/300:26:473
379399257,17cyclictest0-21swapper/221:38:212
379199250,24cyclictest0-21swapper/121:45:021
379399240,21cyclictest17616-21grep00:35:142
379399240,1cyclictest0-21swapper/200:13:272
379199240,5cyclictest0-21swapper/120:05:131
3786992422,1cyclictest9268-21sed21:55:250
3796992323,0cyclictest0-21swapper/321:28:133
379399239,13cyclictest0-21swapper/221:47:402
379399230,23cyclictest0-21swapper/200:23:342
3791992321,1cyclictest0-21swapper/122:35:401
379399228,11cyclictest0-21swapper/222:08:322
3793992214,8cyclictest0-21swapper/200:26:002
3793992211,11cyclictest0-21swapper/223:21:532
3793992211,11cyclictest0-21swapper/223:21:532
3793992211,10cyclictest0-21swapper/222:35:132
379399220,21cyclictest0-21swapper/221:51:012
378699220,20cyclictest0-21swapper/023:56:160
379699214,12cyclictest326-21apt-get20:10:133
379399219,11cyclictest0-21swapper/223:55:132
379399216,2cyclictest0-21swapper/219:25:132
379399215,16cyclictest0-21swapper/200:01:562
379399213,14cyclictest0-21swapper/222:30:142
3793992115,4cyclictest26060-21sed22:25:152
3791992120,1cyclictest0-21swapper/121:41:151
378699218,6cyclictest15638-21perf00:15:000
379399208,7cyclictest31395-31find00:05:192
3793992020,0cyclictest28939-21diskmemload22:17:162
3793992019,1cyclictest28939-21diskmemload21:56:502
3793991918,1cyclictest0-21swapper/223:50:112
3793991917,1cyclictest0-21swapper/222:49:382
379399191,17cyclictest0-21swapper/200:31:532
379399191,15cyclictest0-21swapper/222:57:282
313652190,0sleep30-21swapper/321:30:173
3796991818,0cyclictest0-21swapper/323:20:153
3796991818,0cyclictest0-21swapper/323:20:143
3796991817,1cyclictest0-21swapper/300:39:033
3796991815,2cyclictest0-21swapper/321:17:543
3796991815,1cyclictest0-21swapper/322:43:393
379399185,10cyclictest0-21swapper/223:01:552
379399183,12cyclictest0-21swapper/219:10:122
3793991816,2cyclictest28939-21diskmemload23:11:182
3793991816,2cyclictest28939-21diskmemload23:07:222
3793991813,4cyclictest20225-21kworker/u8:022:10:332
3793991811,6cyclictest0-21swapper/200:15:132
379399180,18cyclictest0-21swapper/221:29:052
3791991814,3cyclictest38150irq/131-snd_hda21:05:211
379399178,7cyclictest0-21swapper/222:24:052
379399178,7cyclictest0-21swapper/221:20:302
379399175,0cyclictest0-21swapper/221:14:552
3793991713,3cyclictest4773-21taskset23:30:052
3793991713,3cyclictest12084-21basename20:34:592
379399170,17cyclictest0-21swapper/222:53:442
3791991717,0cyclictest0-21swapper/122:50:481
3796991616,0cyclictest0-21swapper/323:36:143
379399168,7cyclictest0-21swapper/223:29:472
379399168,7cyclictest0-21swapper/223:29:472
3793991615,1cyclictest28939-21diskmemload22:41:552
3793991615,1cyclictest28939-21diskmemload21:34:442
3793991614,1cyclictest0-21swapper/220:50:132
3793991610,5cyclictest0-21swapper/219:50:012
379399160,13cyclictest0-21swapper/223:45:172
379199163,13cyclictest28939-21diskmemload22:05:571
379199161,7cyclictest28939-21diskmemload23:05:401
379199160,2cyclictest0-21swapper/123:11:141
379199160,12cyclictest0-21swapper/122:32:431
379199160,0cyclictest0-21swapper/100:17:051
378699167,4cyclictest30499-21apache_volume21:30:120
3786991615,1cyclictest13-21ksoftirqd/000:15:260
3786991612,3cyclictest23210-21ls21:25:170
379699159,2cyclictest141rcu_preempt21:55:163
3796991511,3cyclictest5076-21wc19:10:183
379699151,13cyclictest0-21swapper/300:30:323
379699150,1cyclictest0-21swapper/323:49:053
379699150,14cyclictest0-21swapper/300:06:433
379399157,7cyclictest0-21swapper/223:18:222
379399154,7cyclictest0-21swapper/221:16:322
379399151,14cyclictest0-21swapper/222:00:252
3791991514,1cyclictest0-21swapper/123:27:541
3791991514,1cyclictest0-21swapper/123:27:541
3791991512,2cyclictest25-21ksoftirqd/121:24:171
3791991512,2cyclictest0-21swapper/120:00:141
379199150,15cyclictest0-21swapper/122:03:041
378699152,7cyclictest0-21swapper/022:05:140
378699152,3cyclictest141rcu_preempt00:07:330
3786991515,0cyclictest0-21swapper/019:25:130
3786991510,3cyclictest23968-21sleep21:00:000
3796991411,2cyclictest9479-21grep22:15:153
3796991411,2cyclictest23027-21basename19:50:013
3796991410,3cyclictest25771-21grep22:25:143
379699140,2cyclictest0-21swapper/321:41:593
379399147,6cyclictest6123-21cron19:19:552
379399146,7cyclictest0-21swapper/219:55:192
3793991413,1cyclictest0-21swapper/221:40:142
379199141,2cyclictest225-21systemd-journal00:04:591
379199140,13cyclictest29982-21cron21:30:001
378699149,3cyclictest30160-21basename20:05:000
378699146,1cyclictest7065-21df_inode23:50:140
378699145,5cyclictest225-21systemd-journal23:11:320
378699143,3cyclictest12812-21apt-get23:35:020
3786991410,3cyclictest1743-21cut22:30:150
379699139,3cyclictest9999-21ssh22:35:133
3796991310,2cyclictest32373-21cat22:10:023
3796991310,2cyclictest24654-21cut00:20:153
3796991310,2cyclictest13070-21grep19:25:263
379399139,3cyclictest12477-21egrep20:35:152
379399138,4cyclictest29891-21cat23:45:002
379399130,12cyclictest17867-21head20:45:162
379199139,2cyclictest8542-21cat20:25:191
379199138,5cyclictest38150irq/131-snd_hda23:04:461
379199138,3cyclictest4047-21grep20:15:221
379199138,2cyclictest28939-21diskmemload22:11:271
379199135,4cyclictest118542sleep121:18:551
379199131,12cyclictest15695-21ssh23:35:341
3791991310,2cyclictest7094-21cron21:55:001
3791991310,2cyclictest18110-21sed00:35:171
3791991310,2cyclictest16287-21gzip23:55:211
379199130,12cyclictest22352-21ssh23:20:261
379199130,12cyclictest22352-21ssh23:20:261
378699139,3cyclictest6260-21sed21:15:150
378699139,3cyclictest24407-21wc21:45:190
378699139,2cyclictest0-21swapper/023:20:080
378699139,2cyclictest0-21swapper/023:20:080
378699138,3cyclictest18956-21wc23:00:000
378699137,6cyclictest1436-21sendmail-mta23:46:000
378699137,5cyclictest13-21ksoftirqd/022:10:440
378699135,7cyclictest22750-21awk21:45:000
378699131,5cyclictest23905-21/usr/sbin/munin23:40:260
3786991310,2cyclictest24849-21tr00:20:150
3786991310,2cyclictest23818-21df19:50:140
3786991310,2cyclictest23818-21df19:50:140
3786991310,2cyclictest20067-21cron23:20:010
3786991310,2cyclictest19376-21basename23:00:140
3786991310,2cyclictest13-21ksoftirqd/023:26:300
3786991310,2cyclictest13-21ksoftirqd/023:26:300
378699130,12cyclictest11215-21wc20:30:210
379699129,2cyclictest8353-21wc22:15:003
379699129,2cyclictest7325-21grep21:35:153
379699129,2cyclictest29523-21df23:25:143
379699129,2cyclictest29523-21df23:25:143
379699129,2cyclictest2843-21tr22:50:123
379699129,2cyclictest18722-21sed20:45:263
379699129,2cyclictest18230-21cron19:40:003
379699129,2cyclictest13066-21grep20:35:173
379699129,2cyclictest12326-21cut23:15:143
379699129,2cyclictest11206-21tr22:55:133
379699129,2cyclictest10702-21grep20:30:173
379699124,4cyclictest225-21systemd-journal21:15:003
379699121,9cyclictest16773-21grep23:55:263
379699121,7cyclictest16744-21sed00:15:153
3796991212,0cyclictest0-21swapper/323:09:193
379699120,11cyclictest6377-21head23:30:213
379699120,11cyclictest4667-21grep23:10:163
379699120,11cyclictest22460-21cut20:55:163
379399129,2cyclictest3524-21grep20:15:172
379399129,2cyclictest29730-21grep20:00:262
379399129,2cyclictest25666-21ls21:00:232
379399128,3cyclictest0-21swapper/220:29:592
379399122,1cyclictest0-21swapper/220:15:002
379199129,2cyclictest20189-21head20:50:161
379199128,3cyclictest5119-21grep19:10:181
379199128,3cyclictest25910-21grep19:55:151
379199128,3cyclictest13115-21hddtemp_smartct23:15:181
379199127,4cyclictest885-21expr00:05:251
379199127,4cyclictest6923-21expr21:35:131
379199127,4cyclictest17663-21sed20:45:141
379199127,4cyclictest10903-21ssh00:11:191
379199120,12cyclictest0-21swapper/123:42:491
379199120,10cyclictest0-21swapper/123:32:221
378699129,2cyclictest9494-21grep00:30:190
378699129,2cyclictest7335-21tail21:35:150
378699129,2cyclictest4043-21head20:15:220
378699129,2cyclictest32410-21cut21:50:160
378699129,2cyclictest29144-21ntpq23:05:190
378699129,2cyclictest26189-21sort22:25:160
378699129,2cyclictest18198-21sort22:20:170
378699129,2cyclictest13955-21grep20:35:270
378699129,2cyclictest11751-21grep22:35:250
378699129,2cyclictest10600-21grep22:15:250
378699129,2cyclictest10408-21cron22:55:000
378699128,3cyclictest16199-21grep19:35:140
378699126,2cyclictest225-21systemd-journal20:55:010
378699124,2cyclictest30383-21/usr/sbin/munin20:05:150
378699123,6cyclictest14078-21cpuspeed_turbos21:20:130
378699120,11cyclictest25130-21awk00:40:000
379699119,1cyclictest19370-21ssh23:00:143
379699118,2cyclictest9233-21grep19:20:133
379699118,2cyclictest8669-21ssh23:50:353
379699118,2cyclictest4599-21ssh22:31:173
379699118,2cyclictest31132-21grep20:05:173
379699118,2cyclictest27819-21ssh21:47:273
379699118,2cyclictest27780-21basename20:00:003
379699118,2cyclictest26597-21ssh22:45:133
379699118,2cyclictest26033-21ssh00:01:063
379699118,2cyclictest213932sleep320:52:463
379699118,2cyclictest19146-21cut19:40:163
379699118,2cyclictest18465-21ssh22:00:493
379699118,2cyclictest15363-21tr21:20:213
379699118,2cyclictest14926-21cat20:40:123
379699117,3cyclictest15454-21grep19:30:263
379699111,8cyclictest21919-21ssh23:40:123
379699110,10cyclictest18306-21needreboot22:20:173
379399118,2cyclictest28220-21python21:05:252
379399118,2cyclictest25380-21ls19:55:002
379399118,2cyclictest25380-21ls19:55:002
379399118,2cyclictest15651-21ssh23:35:322
379399118,2cyclictest15472-21grep20:40:142
379399118,2cyclictest14950-21grep19:30:202
379399114,4cyclictest0-21swapper/219:35:122
379399113,5cyclictest7507-21ls20:25:022
379399111,7cyclictest0-21swapper/219:20:132
379199119,1cyclictest12921-21users22:55:261
379199118,2cyclictest9957-21grep22:15:181
379199118,2cyclictest6473-21cut23:50:121
379199118,2cyclictest4731-21ssh00:27:121
379199118,2cyclictest30552-21cat23:45:161
379199118,2cyclictest25541-21wget22:25:131
379199118,2cyclictest24292-21ssh00:20:041
379199118,2cyclictest20984-21cut19:45:121
379199118,2cyclictest18890-21cat19:40:141
379199118,2cyclictest14716-21taskset20:40:091
379199118,2cyclictest13979-21grep20:35:271
379199118,2cyclictest11655-21sed20:30:261
379199118,2cyclictest0-21swapper/121:56:361
379199117,3cyclictest29583-21cut21:10:131
379199117,3cyclictest12579-21ls19:25:211
379199116,4cyclictest6641-21cut19:15:131
379199114,2cyclictest9783-21tail19:20:171
379199113,7cyclictest18368-21/usr/sbin/munin22:20:191
379199110,2cyclictest0-21swapper/100:30:411
378699118,2cyclictest9737-21head19:20:170
378699118,2cyclictest8045-21ls20:25:150
378699118,2cyclictest5337-21grep19:10:200
378699118,2cyclictest5315-21cat23:30:140
378699118,2cyclictest29424-21ssh22:46:050
378699118,2cyclictest29235-21ssh21:10:030
378699118,2cyclictest28934-21sh21:10:000
378699118,2cyclictest26633-21wc19:55:190
378699118,2cyclictest24484-21grep21:00:140
378699118,2cyclictest23197-21ls19:50:000
378699118,2cyclictest19406-21ssh22:01:320
378699118,2cyclictest18793-21expr19:40:140
378699118,2cyclictest18094-21ssh22:40:060
378699118,2cyclictest1280-21sed00:25:160
378699118,0cyclictest0-21swapper/000:01:140
378699117,3cyclictest5435-21grep20:20:130
378699117,2cyclictest2050-21sed20:10:270
379699108,1cyclictest7819-21grep20:25:153
379699108,1cyclictest299982sleep320:03:143
379699108,1cyclictest27177-21ls21:05:153
379699108,1cyclictest24075-21hostname19:50:173
379699108,1cyclictest24075-21hostname19:50:173
379699107,2cyclictest7525-21ssh00:10:033
379699107,2cyclictest7329-21head19:15:163
379699107,2cyclictest3130-21expr20:15:153
379699107,2cyclictest25142-21tune2fs21:00:173
379399108,1cyclictest22279-21expr20:55:152
379399107,2cyclictest30388-21cat20:05:112
379399107,2cyclictest19416-21cat19:40:172
379199108,1cyclictest24338-21dpkg21:00:021
379199108,1cyclictest236352sleep120:56:051
379199108,1cyclictest14711-21wc19:30:171
379199107,2cyclictest30331-21ssh21:30:061
379199107,2cyclictest24600-21which19:50:211
379199107,2cyclictest24600-21which19:50:211
379199107,2cyclictest16452-21cat19:35:141
379199100,9cyclictest6550-21tr20:20:211
378699108,1cyclictest6508-21dpkg19:15:000
378699106,2cyclictest15393-21fschecks_time20:40:140
37969997,1cyclictest6015-21cat20:20:173
37919997,1cyclictest569-21cpuspeed_turbos20:10:141
37869997,1cyclictest17429-21ls20:45:140
37869997,1cyclictest15069-21sendmail_mailqu19:30:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional