You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-01 - 21:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed Jul 01, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1074729469,8sleep00-21swapper/007:07:340
1023829482,8sleep20-21swapper/207:05:272
1075528766,6sleep10-21swapper/107:07:391
1060828673,9sleep30-21swapper/307:05:443
11028992825,3cyclictest17023-21kworker/3:111:45:333
1102899280,27cyclictest0-21swapper/311:37:343
1102299280,27cyclictest0-21swapper/112:05:191
11028992724,2cyclictest17023-21kworker/3:110:35:033
1102299270,2cyclictest0-21swapper/112:37:341
11028992624,2cyclictest17023-21kworker/3:111:50:233
11022992623,2cyclictest94-21kswapd009:37:551
11028992524,1cyclictest0-21swapper/310:22:363
1102899252,21cyclictest0-21swapper/310:29:163
11028992511,14cyclictest17023-21kworker/3:112:11:413
1102899250,3cyclictest0-21swapper/309:23:053
1102899250,24cyclictest0-21swapper/310:42:143
11028992424,0cyclictest0-21swapper/310:07:343
1102299240,2cyclictest0-21swapper/109:49:111
11028992320,3cyclictest17023-21kworker/3:112:08:433
11028992319,4cyclictest17023-21kworker/3:110:48:433
1102299230,3cyclictest0-21swapper/109:25:141
1102299220,1cyclictest0-21swapper/110:59:381
11022992119,1cyclictest0-21swapper/111:20:121
1102299210,3cyclictest0-21swapper/110:53:281
11028992017,3cyclictest17023-21kworker/3:111:08:533
11028992015,5cyclictest17023-21kworker/3:109:25:333
1102899200,19cyclictest17023-21kworker/3:109:40:223
1102899200,18cyclictest0-21swapper/310:55:413
1102299200,2cyclictest0-21swapper/110:49:281
1102299200,0cyclictest0-21swapper/111:58:091
1101899197,6cyclictest6158-21perf09:50:010
11028991815,3cyclictest17023-21kworker/3:110:50:533
11026991817,1cyclictest0-21swapper/212:35:272
1102299180,1cyclictest0-21swapper/112:26:291
1102299180,1cyclictest0-21swapper/110:31:441
11018991818,0cyclictest0-21swapper/011:33:120
11028991715,2cyclictest17023-21kworker/3:111:56:233
11028991713,3cyclictest11790-21ssh09:33:503
1102699170,17cyclictest0-21swapper/212:08:452
11022991713,4cyclictest0-21swapper/110:24:071
1102299170,17cyclictest0-21swapper/110:17:031
1102299170,13cyclictest0-21swapper/111:15:031
1101899170,2cyclictest0-21swapper/009:11:050
11028991615,1cyclictest17023-21kworker/3:110:12:133
11028991614,2cyclictest17023-21kworker/3:111:44:023
11028991614,2cyclictest17023-21kworker/3:109:52:133
11028991614,2cyclictest17023-21kworker/3:109:48:533
11028991613,3cyclictest17023-21kworker/3:110:32:033
1102699161,14cyclictest13603-21/usr/sbin/munin07:15:172
11022991616,0cyclictest0-21swapper/112:10:111
11022991615,1cyclictest0-21swapper/111:40:371
1102299160,2cyclictest0-21swapper/110:25:231
1102299160,1cyclictest0-21swapper/109:22:581
1102299160,15cyclictest0-21swapper/109:44:421
1101899169,6cyclictest27533-21apache212:32:010
1101899160,1cyclictest0-21swapper/009:40:450
1102699159,5cyclictest25258-21sed10:00:252
11026991512,2cyclictest0-21swapper/209:56:592
11026991511,3cyclictest21639-21ssh09:20:162
1102699150,1cyclictest0-21swapper/211:16:452
1102699150,1cyclictest0-21swapper/209:10:072
1102699150,15cyclictest0-21swapper/210:49:042
1102299154,5cyclictest31834-21ssh10:05:131
11022991514,1cyclictest0-21swapper/109:18:161
11022991513,2cyclictest38150irq/131-snd_hda12:00:251
11022991513,2cyclictest38150irq/131-snd_hda12:00:251
1101899157,7cyclictest10083-21munin-run10:50:010
1101899150,9cyclictest17163-21basename08:30:000
1101899150,14cyclictest0-21swapper/011:03:520
60042140,1sleep00-21swapper/012:23:020
11028991410,3cyclictest18648-21tr07:25:163
11028991410,3cyclictest14632-21grep09:15:273
11028991410,3cyclictest1213-21grep12:20:163
1102699149,5cyclictest27533-21apache209:00:182
1102699141,11cyclictest20522-21latency_hist07:30:002
11026991410,3cyclictest20499-21grep08:35:182
11026991410,3cyclictest17731-21tr12:30:142
1102699140,1cyclictest0-21swapper/207:50:142
1102299149,4cyclictest13719-21wc08:20:201
1102299143,4cyclictest225-21systemd-journal07:40:341
1102299143,4cyclictest225-21systemd-journal07:40:341
11022991410,3cyclictest28397-21grep11:00:161
11022991410,2cyclictest17442-21grep08:30:151
1102299140,2cyclictest0-21swapper/111:11:161
1102299140,1cyclictest0-21swapper/111:50:411
1102299140,1cyclictest0-21swapper/111:36:491
1102299140,13cyclictest0-21swapper/111:46:371
1102299140,0cyclictest0-21swapper/109:53:291
1101899148,5cyclictest27531-21apache210:50:060
1101899147,6cyclictest30923-21sendmail-msp12:00:000
11018991412,2cyclictest13-21ksoftirqd/011:08:300
11018991410,2cyclictest32604-21grep10:05:170
1101899140,1cyclictest0-21swapper/010:21:030
1102899138,4cyclictest12252-21aten2_r7power_c11:10:133
1102899135,5cyclictest0-21swapper/308:50:153
1102899132,6cyclictest17431-21apache_accesses12:30:113
11028991310,2cyclictest6325-21wc08:05:213
11028991310,2cyclictest29747-21grep11:20:163
11028991310,2cyclictest26806-21wc08:50:003
11028991310,2cyclictest24105-21grep10:00:163
1102699139,3cyclictest5280-21ssh11:05:212
1102699139,3cyclictest1123-21seq10:25:162
1102699136,6cyclictest25047-21tr10:20:152
1102699133,5cyclictest225-21systemd-journal09:20:002
11026991312,1cyclictest0-21swapper/212:00:192
11026991312,1cyclictest0-21swapper/212:00:192
11026991310,2cyclictest9386-21grep12:25:152
11026991310,2cyclictest5806-21basename09:30:272
11026991310,2cyclictest25640-21cron10:40:002
11026991310,2cyclictest25584-21grep07:40:142
11026991310,2cyclictest25584-21grep07:40:142
11026991310,2cyclictest25419-21tail12:15:162
11026991310,2cyclictest24113-21wc07:35:192
11026991310,2cyclictest23985-21basename10:19:592
11026991310,2cyclictest17157-21basename08:30:002
11026991310,2cyclictest14482-21sh09:55:002
11026991310,2cyclictest10801-21grep10:30:232
1102699130,12cyclictest25216-21grep08:45:172
1102699130,12cyclictest0-21swapper/210:54:022
1102299139,3cyclictest15315-21grep07:15:271
1102299139,2cyclictest27235-21wc10:40:171
1102299139,2cyclictest25276-21turbostat07:40:001
11022991310,2cyclictest7178-21proc_pri09:30:341
11022991310,2cyclictest30108-21cut08:55:191
11022991310,2cyclictest17513-21tr12:30:131
1102299130,1cyclictest0-21swapper/111:32:441
1102299130,12cyclictest6931-21grep08:05:291
1102299130,0cyclictest0-21swapper/111:06:251
1101899139,3cyclictest22489-21grep09:20:210
1101899138,3cyclictest8863-21ls08:10:220
11018991310,2cyclictest27191-21tr08:50:160
11018991310,2cyclictest26205-21tail08:45:250
1101899130,7cyclictest23930-21awk12:15:000
1101899130,1cyclictest0-21swapper/010:25:350
1101899130,13cyclictest0-21swapper/010:56:270
1101899130,13cyclictest0-21swapper/010:18:270
1102899129,3cyclictest17023-21kworker/3:111:30:313
1102899129,2cyclictest4683-21cat09:10:153
1102899129,2cyclictest28761-21head07:45:173
1102899129,2cyclictest24902-21grep12:15:143
1102899129,2cyclictest14517-21grep09:35:163
1102899129,2cyclictest11483-21grep07:10:143
11028991211,1cyclictest0-21swapper/312:02:413
11028991211,1cyclictest0-21swapper/312:02:413
1102699129,2cyclictest32606-21sort10:05:162
1102699129,2cyclictest28442-21cut11:00:162
1102699129,2cyclictest2384-21sort09:05:172
1102699129,2cyclictest22456-21grep08:40:152
1102699129,2cyclictest22362-21ssh09:40:152
1102699129,2cyclictest21526-21sed07:30:162
1102699129,2cyclictest18497-21cut08:30:202
1102699129,2cyclictest16761-21grep07:20:172
1102699129,2cyclictest14033-21cron11:50:002
1102699129,2cyclictest13045-21tr11:10:172
1102699129,2cyclictest10357-21grep08:15:122
1102699128,3cyclictest31981-21ls09:45:212
1102699125,3cyclictest11008-21cyclictest10:59:222
1102699122,8cyclictest29620-21fschecks_count11:20:162
11026991210,1cyclictest0-21swapper/211:42:222
1102699120,2cyclictest4928-21ssh11:25:112
1102299129,2cyclictest3466-21grep08:00:161
1102299129,2cyclictest22194-21wc07:30:241
1102299129,2cyclictest2062-21cut12:20:221
1102299127,4cyclictest20143-21ssh10:35:441
1102299127,2cyclictest225-21systemd-journal11:27:041
1101899129,2cyclictest5566-21tr08:05:170
1101899129,2cyclictest3831-21basename09:10:000
1101899129,2cyclictest27531-21apache211:26:430
1101899129,2cyclictest26817-21grep10:40:150
1101899129,2cyclictest23858-21sed08:40:270
1101899129,2cyclictest23467-21cut10:00:130
1101899129,2cyclictest20507-21tail08:35:180
1101899129,2cyclictest19181-21head10:35:230
1101899129,2cyclictest17639-21grep07:20:260
1101899129,2cyclictest10833-21ssh10:30:220
1101899129,2cyclictest0-21swapper/009:26:330
1101899128,3cyclictest8650-21cut12:05:180
1101899128,2cyclictest11289-21grep07:10:140
1101899123,7cyclictest0-21swapper/012:36:400
1102899118,2cyclictest8409-21head08:10:173
1102899118,2cyclictest3937-21cut08:00:193
1102899118,2cyclictest32374-21grep09:00:173
1102899118,2cyclictest30946-21ssh12:37:533
1102899118,2cyclictest27438-21ssh11:00:073
1102899118,2cyclictest26369-21seq11:18:213
1102899118,2cyclictest23701-21grep07:35:173
1102899118,2cyclictest21250-21cat08:35:263
1102899118,2cyclictest19492-21chrt08:34:513
1102899118,2cyclictest16381-21cut10:15:143
1102899118,2cyclictest13399-21head08:20:183
1102899118,2cyclictest12897-21ssh12:26:493
1102899118,2cyclictest10494-21grep08:15:133
1102899117,3cyclictest23410-21tr08:40:223
1102899115,3cyclictest17023-21kworker/3:109:55:513
1102899110,10cyclictest9992-21ssh11:27:493
1102699118,2cyclictest29528-21basename09:25:142
1102699118,2cyclictest29401-21ssh10:41:212
1102699118,2cyclictest29331-21cat08:55:142
1102699118,2cyclictest26993-21grep08:50:152
1102699118,2cyclictest24942-21ssh11:55:262
1102699118,2cyclictest23039-21ssh11:35:202
1102699118,2cyclictest21332-21ssh09:39:442
1102699118,2cyclictest18376-21ssh11:51:522
1102699118,2cyclictest17818-21ssh11:32:222
1102699118,2cyclictest15934-21sh12:10:082
1102699118,2cyclictest1478-21grep12:20:172
1102699118,2cyclictest10776-21ssh10:11:172
1102299118,2cyclictest8928-21ps08:10:221
1102299118,2cyclictest28832-21cat07:45:181
1102299118,2cyclictest28680-21grep08:50:281
1102299118,2cyclictest26499-21chrt08:48:081
1102299118,2cyclictest17844-21sh09:56:061
1102299118,2cyclictest1462-21basename09:05:001
1102299118,1cyclictest8223-21kworker/1:107:30:001
1102299117,3cyclictest25216-21sed12:15:151
1102299114,3cyclictest225-21systemd-journal08:40:001
1102299113,5cyclictest1-21systemd10:11:341
1102299110,2cyclictest0-21swapper/110:00:281
1102299110,1cyclictest0-21swapper/107:50:141
1102299110,11cyclictest0-21swapper/108:25:151
1102299110,10cyclictest16886-21if_enp0s31f607:20:181
1101899118,2cyclictest9256-21ssh11:46:180
1101899118,2cyclictest6524-21ssh09:30:300
1101899118,2cyclictest3293-21expr08:00:150
1101899118,2cyclictest32733-21date07:55:010
1101899118,2cyclictest31140-21ssh12:00:010
1101899118,2cyclictest31140-21ssh12:00:010
1101899118,2cyclictest30095-21ssh11:40:100
1101899118,2cyclictest25933-21expr07:40:150
1101899118,2cyclictest25933-21expr07:40:150
1101899118,2cyclictest25790-21ssh12:15:200
1101899118,2cyclictest24647-21chrt11:17:020
1101899118,2cyclictest22995-21awk11:35:200
1101899118,2cyclictest21162-21ssh09:39:370
1101899118,2cyclictest19780-21ssh09:19:250
1101899118,2cyclictest15854-21ssh09:55:160
1101899118,2cyclictest15389-21tail11:50:160
1101899118,2cyclictest14097-21ssh11:10:250
1101899118,2cyclictest12883-21grep08:20:150
1101899118,2cyclictest11395-21rm09:52:360
1101899118,2cyclictest11241-21ssh10:11:380
1101899117,3cyclictest9895-21head12:25:180
1101899117,2cyclictest734-21grep07:55:150
1101899114,3cyclictest1449-21munin-plugin-st09:05:000
1101899113,6cyclictest25279-21perf07:40:000
11018991110,1cyclictest0-21swapper/011:20:120
1101899110,10cyclictest30092-21head08:55:200
1102899108,1cyclictest225752sleep307:31:093
1102899108,1cyclictest16628-21fschecks_count07:20:163
1102899108,1cyclictest13907-21expr07:15:153
1102899108,1cyclictest1328-21hostname07:55:183
1102899107,2cyclictest29551-21ls08:55:173
1102899107,2cyclictest26506-21cat07:40:193
1102899107,2cyclictest26506-21cat07:40:193
1102899107,2cyclictest15041-21dpkg08:25:143
1102899105,3cyclictest3061-21mailstats09:05:213
1102699108,1cyclictest8650-21latency08:10:192
1102699108,1cyclictest6651-21which08:05:252
1102699108,1cyclictest3864-21cut08:00:192
1102699108,1cyclictest2272-21chrt07:55:322
1102699108,1cyclictest13116-21taskset07:11:592
1102699107,2cyclictest28632-21sort07:45:172
1102299108,1cyclictest11897-21grep07:10:161
1102299107,2cyclictest5381-21uname09:10:191
1102299105,2cyclictest225-21systemd-journal07:57:031
1101899108,1cyclictest29893-21taskset07:47:370
1101899108,1cyclictest22667-21taskset07:32:210
1101899108,1cyclictest17716-21cstates08:30:160
1101899108,1cyclictest156172sleep007:18:320
1101899107,2cyclictest10714-21ls08:15:130
110289996,2cyclictest30898-21cat07:50:153
110269997,1cyclictest14425-21taskset08:20:432
110229997,1cyclictest3603-21taskset09:07:261
110229997,1cyclictest12240-21taskset08:18:261
110229996,2cyclictest22423-21cut08:40:151
110189996,2cyclictest18716-21cstates07:25:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional