You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-20 - 08:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Sun Sep 20, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
752029265,16sleep10-21swapper/119:09:551
724328976,9sleep00-21swapper/019:06:220
719028976,9sleep20-21swapper/219:05:382
749728371,8sleep30-21swapper/319:09:373
761899270,27cyclictest0-21swapper/221:21:332
7616992423,1cyclictest0-21swapper/122:43:591
761899230,2cyclictest0-21swapper/222:40:142
761899230,23cyclictest0-21swapper/223:13:352
761699230,19cyclictest0-21swapper/122:15:141
761899220,22cyclictest0-21swapper/222:35:532
7621992114,2cyclictest39-21ksoftirqd/300:38:163
7616992121,0cyclictest0-21swapper/121:51:221
256842210,1sleep20-21swapper/222:22:292
7611992019,1cyclictest0-21swapper/000:07:160
761899191,18cyclictest0-21swapper/221:10:172
761199190,0cyclictest0-21swapper/023:05:450
7618991816,1cyclictest32-21ksoftirqd/221:35:172
761899180,17cyclictest0-21swapper/200:10:012
761899180,16cyclictest0-21swapper/221:42:172
7616991816,1cyclictest9892-21latency_hist23:50:001
762199170,0cyclictest0-21swapper/322:34:193
761899170,17cyclictest0-21swapper/223:09:342
761899170,15cyclictest0-21swapper/222:00:182
7616991715,1cyclictest0-21swapper/121:47:061
7616991714,2cyclictest25184-21sed20:50:251
761199170,7cyclictest0-21swapper/023:25:140
43242170,0sleep00-21swapper/022:29:360
7621991613,2cyclictest13682-21ls19:20:193
7616991614,1cyclictest4028-21grep00:05:171
7616991614,1cyclictest0-21swapper/123:05:061
7616991611,5cyclictest0-21swapper/122:37:231
761699160,16cyclictest0-21swapper/121:26:111
7611991611,3cyclictest13-21ksoftirqd/022:51:150
7621991514,1cyclictest0-21swapper/322:20:383
761899151,1cyclictest141rcu_preempt21:17:122
761899150,2cyclictest0-21swapper/223:23:482
761899150,15cyclictest0-21swapper/223:38:012
761899150,14cyclictest0-21swapper/222:13:042
761699159,5cyclictest13331-21head19:20:171
761699150,15cyclictest0-21swapper/121:44:431
761199155,3cyclictest0-21swapper/021:20:160
7611991515,0cyclictest0-21swapper/022:41:390
761199150,1cyclictest141rcu_preempt21:17:260
762199143,6cyclictest15429-21sh23:14:593
762199142,11cyclictest0-21swapper/322:00:243
7621991413,1cyclictest0-21swapper/323:54:533
7621991411,2cyclictest17000-21cut20:35:173
762199140,14cyclictest0-21swapper/321:27:243
762199140,14cyclictest0-21swapper/300:29:273
761899149,3cyclictest9756-21idleruntime-cro21:34:592
761899141,1cyclictest141rcu_preempt22:30:202
7618991411,2cyclictest8960-21cron20:20:002
7618991411,2cyclictest25175-21grep23:00:262
761899140,6cyclictest0-21swapper/222:07:122
761699145,4cyclictest31137-21munin-run23:04:591
7616991414,0cyclictest0-21swapper/122:48:571
761699140,9cyclictest0-21swapper/100:35:161
761699140,1cyclictest0-21swapper/119:35:141
761699140,14cyclictest0-21swapper/122:09:181
761699140,14cyclictest0-21swapper/100:34:401
761199146,7cyclictest19639-21munin-run00:15:010
761199145,0cyclictest141rcu_preempt22:14:400
7611991411,2cyclictest0-21swapper/022:32:410
7611991410,3cyclictest8735-21ssh21:53:120
762199139,3cyclictest16103-21cron20:35:003
762199137,5cyclictest24431-21sort23:00:193
762199137,5cyclictest18142-21apache223:34:383
762199134,7cyclictest29410-21apt-get22:25:133
762199132,2cyclictest0-21swapper/323:45:513
7621991311,1cyclictest7603-21cyclictest22:50:133
7621991310,3cyclictest39-21ksoftirqd/323:41:013
7621991310,2cyclictest8672-21ssh21:34:103
7621991310,2cyclictest31169-21wc19:55:273
7621991310,2cyclictest30226-21sed22:05:263
762199130,8cyclictest0-21swapper/300:02:223
762199130,2cyclictest0-21swapper/322:40:403
762199130,13cyclictest0-21swapper/300:08:353
762199130,12cyclictest12670-21cut22:15:143
762199130,0cyclictest0-21swapper/321:17:043
761899139,3cyclictest17847-21grep19:30:152
761899139,2cyclictest16876-21tail19:25:282
761899138,5cyclictest18143-21apache221:58:232
761899136,4cyclictest537-21snmpd00:22:442
761899133,4cyclictest225-21systemd-journal22:53:232
7618991311,2cyclictest32-21ksoftirqd/222:15:102
7618991310,2cyclictest6099-21grep20:10:252
7618991310,2cyclictest4065-21head21:50:182
7618991310,2cyclictest32-21ksoftirqd/220:00:012
7618991310,2cyclictest30459-21dump-pmu-power21:05:002
7618991310,2cyclictest28032-21grep21:45:172
7618991310,2cyclictest27373-21grep23:40:212
7618991310,2cyclictest22892-21tr19:40:162
7618991310,2cyclictest22345-21idleruntime-cro19:39:592
761899130,1cyclictest0-21swapper/223:56:312
761899130,1cyclictest0-21swapper/223:56:302
761899130,12cyclictest27662-21grep20:55:272
761699139,3cyclictest14224-21tr20:30:161
761699139,2cyclictest16017-21ssh23:53:151
761699135,7cyclictest0-21swapper/122:31:501
7616991313,0cyclictest0-21swapper/123:28:411
7616991312,1cyclictest38150irq/131-snd_hda21:31:191
7616991310,2cyclictest21920-21tail22:20:171
761699130,12cyclictest5357-21ssh00:25:181
761699130,12cyclictest0-21swapper/123:23:271
761199139,3cyclictest8780-21basename21:15:000
7611991310,2cyclictest4240-21wc20:10:010
7611991310,2cyclictest21933-21grep22:20:170
7611991310,2cyclictest19076-21grep21:40:140
7611991310,2cyclictest15479-21grep19:25:170
7611991310,2cyclictest13882-21wc20:30:000
7611991310,1cyclictest0-21swapper/022:57:250
762199129,2cyclictest8596-21grep19:10:213
762199129,2cyclictest32331-21sed21:05:273
762199129,2cyclictest3225-21basename21:50:143
762199129,2cyclictest3205-21grep20:05:243
762199129,2cyclictest30002-21grep21:00:283
762199129,2cyclictest26658-21grep20:55:183
762199129,2cyclictest21006-21sed19:35:183
762199129,2cyclictest1668-21cut21:10:173
762199129,2cyclictest16303-21cut23:15:143
762199129,2cyclictest15010-21awk00:30:273
762199128,3cyclictest26768-21grep21:44:593
762199128,3cyclictest15280-21cut19:25:163
762199122,9cyclictest18142-21apache221:47:183
762199121,10cyclictest25489-21grep23:20:183
7621991210,1cyclictest11982-21sed21:35:273
762199120,11cyclictest28334-21wc19:50:193
762199120,11cyclictest22689-21chrt21:22:483
762199120,11cyclictest0-21swapper/319:40:153
761899129,2cyclictest30789-21grep21:05:152
761899129,2cyclictest28050-21sort19:50:172
761899129,2cyclictest2752-21cut20:05:202
761899129,2cyclictest18321-21ssh23:55:012
761899129,2cyclictest16513-21grep20:35:152
761899129,2cyclictest14763-21ls19:24:592
761899129,2cyclictest10773-21grep19:15:172
761899128,3cyclictest13237-21tr00:30:142
7618991210,1cyclictest0-21swapper/223:32:032
761899120,1cyclictest0-21swapper/221:25:052
761899120,11cyclictest29706-21grep22:25:152
761899120,11cyclictest25771-21head19:45:192
761699129,2cyclictest31465-21tr21:05:171
761699129,2cyclictest26840-21sed23:40:161
761699129,2cyclictest24266-21grep23:20:001
761699129,2cyclictest19665-21cat22:00:001
761699129,2cyclictest10379-21ssh21:15:201
761699127,2cyclictest11285-21taskset20:24:111
761699127,2cyclictest11285-21taskset20:24:101
761699123,6cyclictest537-21snmpd23:32:321
761699121,11cyclictest0-21swapper/123:59:211
761699121,11cyclictest0-21swapper/123:59:201
761699121,10cyclictest0-21swapper/119:10:181
7616991210,1cyclictest0-21swapper/122:02:031
761699120,11cyclictest2686-21munin-run00:05:011
761199129,2cyclictest9984-21head20:20:210
761199129,2cyclictest9984-21head20:20:200
761199129,2cyclictest944-21grep20:00:220
761199129,2cyclictest8951-21grep19:10:230
761199129,2cyclictest4187-21munin-run00:25:000
761199129,2cyclictest31822-21chrt21:28:260
761199129,2cyclictest31476-21sed21:05:170
761199129,2cyclictest30814-21sh22:45:150
761199129,2cyclictest28129-21head00:00:200
761199129,2cyclictest2785-21head23:45:180
761199129,2cyclictest26773-21ssh00:37:540
761199129,2cyclictest26655-21grep23:40:170
761199129,2cyclictest26215-21grep20:55:160
761199129,2cyclictest21478-21ssh22:00:210
761199129,2cyclictest14917-21cat19:24:590
761199129,2cyclictest14781-21grep20:30:180
761199129,2cyclictest13535-21ssh00:30:150
761199129,2cyclictest12342-21sh19:20:000
761199129,2cyclictest11867-21uname23:50:220
761199127,1cyclictest0-21swapper/023:20:300
7611991210,1cyclictest17823-21grep23:35:140
761199120,11cyclictest27922-21grep19:50:140
762199119,1cyclictest5169-21tail22:10:213
762199119,1cyclictest32012sleep323:06:593
762199119,1cyclictest15671-21tune2fs22:55:183
762199119,1cyclictest0-21swapper/323:35:133
762199118,2cyclictest405-21grep20:00:173
762199118,2cyclictest32003-21ssh22:45:213
762199118,2cyclictest27093-21sh00:19:183
762199118,2cyclictest25827-21date20:55:013
762199118,2cyclictest20814-21head23:55:283
762199118,2cyclictest20814-21head23:55:273
762199118,2cyclictest19257-21grep19:30:263
762199118,2cyclictest19054-21grep20:40:153
762199118,2cyclictest1530-21sed23:25:173
762199118,2cyclictest1487-21ssh00:22:553
762199118,2cyclictest14531-21ssh00:11:013
762199118,2cyclictest14148-21sh21:55:543
762199118,2cyclictest11009-21cut19:15:173
762199118,2cyclictest10520-21wc20:20:263
762199118,2cyclictest10520-21wc20:20:253
762199117,3cyclictest15563-21wc22:35:253
762199115,2cyclictest225-21systemd-journal20:50:003
761899118,2cyclictest9924-21grep20:20:202
761899118,2cyclictest9924-21grep20:20:192
761899118,2cyclictest6053-21sh23:47:102
761899118,2cyclictest5942-21grep00:25:222
761899118,2cyclictest5702-21ssh00:05:502
761899118,2cyclictest4201-21ssh22:48:322
761899118,2cyclictest26987-21sh00:00:022
761899118,2cyclictest24831-21sh00:36:232
761899118,2cyclictest24788-21chrt20:50:202
761899118,2cyclictest23378-21ssh00:16:242
761899118,2cyclictest21923-21ssh23:18:212
761899118,2cyclictest16280-21grep22:55:232
761899118,2cyclictest14757-21head20:30:182
761899118,2cyclictest1395-21uniq23:25:162
761699118,2cyclictest4850-21expr20:10:151
761699118,2cyclictest26020-21grep20:55:151
761699118,2cyclictest23649-21missed_timers19:40:211
761699118,2cyclictest13117-21ntpq00:10:221
761699117,1cyclictest16961-21ssh22:55:291
761699110,11cyclictest0-21swapper/122:10:111
761199118,2cyclictest9518-21grep23:10:280
761199118,2cyclictest8979-21basename20:20:000
761199118,2cyclictest6302-21rm21:32:170
761199118,2cyclictest5507-21tail00:25:190
761199118,2cyclictest4936-21expr20:10:150
761199118,2cyclictest29154-21head22:05:180
761199118,2cyclictest25183-21grep23:00:270
761199118,2cyclictest24054-21df20:50:140
761199118,2cyclictest23123-21taskset20:47:460
761199118,2cyclictest23063-21expr19:40:170
761199118,2cyclictest21183-21ssh23:55:410
761199118,2cyclictest21183-21ssh23:55:410
761199118,2cyclictest16212-21ssh23:15:130
761199118,2cyclictest13314-21sort22:15:170
761199118,2cyclictest10117-21ssh21:35:060
761199117,3cyclictest15418-21fgrep22:35:220
761199115,2cyclictest225-21systemd-journal19:35:000
761199111,9cyclictest28054-21lockfile-create00:20:010
761199110,10cyclictest15302-21ssh21:56:480
761199110,0cyclictest0-21swapper/021:46:390
762199107,2cyclictest6942-21cut20:15:153
762199107,2cyclictest5563-21cut20:10:193
762199107,2cyclictest25110-21cut19:45:163
762199107,2cyclictest11883-21tr20:25:133
761899107,2cyclictest7877-21cut19:10:182
761899107,2cyclictest20657-21chrt20:41:362
761899107,2cyclictest1396-21sed20:00:262
761899107,2cyclictest12881-21wc20:25:202
761899106,3cyclictest22647-21wc20:45:252
761699108,2cyclictest3502-21sh20:05:291
761699108,1cyclictest7374-21taskset23:10:041
761699108,1cyclictest25386-21expr19:45:171
761699108,1cyclictest230142sleep120:46:231
761699108,1cyclictest20709-21rm21:21:161
761699108,1cyclictest19106-21grep23:35:211
761699108,1cyclictest12388-21tail20:25:151
761699107,2cyclictest862-21ntp_states20:00:211
761699107,2cyclictest7436-21expr22:50:181
761699107,2cyclictest32588-21ssh22:26:201
761699107,2cyclictest29490-21turbostat19:55:001
761699107,2cyclictest28651-21ssh00:20:091
761699107,2cyclictest24175-21ssh00:17:011
761699107,2cyclictest15486-21tr19:25:171
761699107,2cyclictest1306-21rm21:10:161
761699107,2cyclictest11530-21strings21:35:211
761699107,2cyclictest10426-21cat19:15:161
761699100,9cyclictest29676-21which21:00:241
761699100,10cyclictest8735-21seq20:17:091
761199108,1cyclictest8897-21sh23:30:080
761199108,1cyclictest29742-21basename19:55:140
761199108,1cyclictest25823-21basename19:45:200
761199108,1cyclictest20432-21grep20:40:260
761199107,2cyclictest16972-21fschecks_count20:35:170
761199100,9cyclictest30315-21sshd21:03:230
761199100,9cyclictest21182-21grep19:35:190
76169998,1cyclictest19697-21cat20:40:181
76169990,8cyclictest17876-21cstates19:30:161
76169990,4cyclictest225-21systemd-journal20:00:001
76169987,1cyclictest17529-21ntpq20:35:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional