You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-03 - 07:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Fri Jul 03, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28578210576,9sleep00-21swapper/019:06:540
2855429667,10sleep30-21swapper/319:06:373
2875028866,6sleep10-21swapper/119:09:081
2855828667,8sleep20-21swapper/219:06:402
2891099252,2cyclictest141rcu_preempt23:49:142
2890499250,22cyclictest0-21swapper/023:30:130
28912992416,5cyclictest39-21ksoftirqd/321:46:323
2891299241,2cyclictest0-21swapper/323:40:033
28912992220,1cyclictest0-21swapper/321:40:133
2891099220,1cyclictest0-21swapper/223:39:592
2890799213,17cyclictest0-21swapper/123:15:171
2890799210,0cyclictest0-21swapper/122:40:421
2891299200,3cyclictest0-21swapper/322:51:413
2890799204,1cyclictest21916-21diskmemload22:10:371
28910991917,2cyclictest0-21swapper/223:57:502
28910991914,4cyclictest32-21ksoftirqd/200:00:222
2891099190,17cyclictest0-21swapper/200:31:362
28904991918,1cyclictest0-21swapper/021:34:250
2890499190,19cyclictest0-21swapper/022:28:360
2890499190,0cyclictest0-21swapper/020:35:130
2891299180,17cyclictest0-21swapper/322:47:193
2891299180,16cyclictest0-21swapper/322:29:513
2891099180,1cyclictest0-21swapper/222:45:142
2891099180,1cyclictest0-21swapper/222:05:512
28907991815,2cyclictest0-21swapper/123:21:261
2890799180,17cyclictest0-21swapper/122:33:491
28904991816,1cyclictest13-21ksoftirqd/019:45:150
2890499180,18cyclictest0-21swapper/021:10:120
28910991713,1cyclictest32-21ksoftirqd/221:58:552
28907991713,3cyclictest25-21ksoftirqd/100:35:221
2890799170,1cyclictest0-21swapper/123:41:191
2890799170,16cyclictest0-21swapper/123:10:531
2891299167,5cyclictest0-21swapper/322:09:593
28912991615,1cyclictest0-21swapper/323:25:143
2891099169,2cyclictest32-21ksoftirqd/223:12:022
2891099161,0cyclictest141rcu_preempt23:16:162
2891099160,1cyclictest141rcu_preempt21:42:072
2891099160,16cyclictest0-21swapper/221:38:522
2891099160,15cyclictest0-21swapper/200:38:542
28907991615,1cyclictest0-21swapper/100:20:281
28907991615,1cyclictest0-21swapper/100:20:271
28907991615,1cyclictest0-21swapper/100:17:001
2890799161,1cyclictest13959-21ssh23:58:311
2890799160,15cyclictest0-21swapper/100:25:581
2890499164,6cyclictest0-21swapper/022:02:230
28904991615,1cyclictest0-21swapper/000:10:520
2890499160,1cyclictest0-21swapper/022:09:320
28910991511,1cyclictest32-21ksoftirqd/222:03:492
2891099150,5cyclictest0-21swapper/200:07:002
2891099150,15cyclictest0-21swapper/223:07:132
2890799153,12cyclictest21916-21diskmemload21:39:441
28907991514,1cyclictest0-21swapper/123:50:301
28907991513,1cyclictest0-21swapper/121:19:231
28907991511,3cyclictest8868-21ssh00:15:001
2890799150,15cyclictest0-21swapper/100:04:171
2890499155,5cyclictest4102-21ssh22:55:010
28904991514,1cyclictest0-21swapper/021:51:100
2890499150,9cyclictest0-21swapper/020:20:150
2890499150,15cyclictest0-21swapper/000:00:320
2890499150,14cyclictest0-21swapper/021:35:520
28912991411,2cyclictest24570-21grep20:05:263
2891299140,13cyclictest0-21swapper/322:37:013
2891099145,6cyclictest7958-21munin-node21:19:082
28910991411,2cyclictest32-21ksoftirqd/222:35:022
28910991410,3cyclictest11028-21grep20:45:172
28910991410,2cyclictest3369-21tr20:30:152
28907991414,0cyclictest0-21swapper/121:28:131
2890799141,13cyclictest0-21swapper/121:31:101
2890799140,4cyclictest0-21swapper/122:06:011
2890799140,4cyclictest0-21swapper/121:50:231
2890499143,6cyclictest21922-21munin-run21:09:590
28904991410,2cyclictest29274-21grep19:10:160
2890499140,10cyclictest0-21swapper/023:40:020
2890499140,0cyclictest0-21swapper/022:10:550
135902140,2sleep10-21swapper/122:21:221
2891299139,3cyclictest28281-21sort00:25:303
2891299139,3cyclictest1552-21sed23:50:283
2891299139,2cyclictest31180-21grep21:15:163
28912991312,1cyclictest39-21ksoftirqd/300:31:513
28912991310,2cyclictest30552-21cat19:10:283
28912991310,2cyclictest29435-21cat20:15:243
28912991310,2cyclictest20511-21head21:05:183
28912991310,2cyclictest1841-21cut21:55:183
28912991310,2cyclictest16304-21grep19:50:153
28912991310,2cyclictest15713-21taskset00:18:503
28912991310,2cyclictest13436-21sed20:50:163
2891299130,12cyclictest0-21swapper/319:55:133
2891099139,3cyclictest17490-21grep21:00:142
2891099139,3cyclictest14753-21cron20:54:592
2891099135,3cyclictest141rcu_preempt23:25:302
28910991310,2cyclictest27404-21if_enp2s000:25:222
28910991310,2cyclictest18833-21head00:20:182
28910991310,2cyclictest18833-21head00:20:172
28910991310,2cyclictest16594-21head19:50:172
28910991310,2cyclictest16327-21ssh23:40:192
28910991310,2cyclictest13732-21grep22:40:252
2891099130,12cyclictest31319-21cat19:15:002
2890799139,4cyclictest38150irq/131-snd_hda23:48:411
2890799139,3cyclictest2624-21basename20:30:001
2890799139,3cyclictest25530-21grep00:05:181
2890799139,3cyclictest21825-21expr23:05:141
2890799138,4cyclictest5580-21basename20:35:141
2890799138,4cyclictest25308-21cat20:10:111
2890799132,7cyclictest25-21ksoftirqd/122:47:131
2890799131,1cyclictest141rcu_preempt21:41:141
28907991311,1cyclictest25-21ksoftirqd/122:25:341
28907991310,2cyclictest16646-21sed20:55:261
28907991310,2cyclictest10295-21cron00:35:011
2890799130,12cyclictest7027-21ssh21:20:151
2890799130,12cyclictest307-21wc19:15:221
2890799130,12cyclictest307-21wc19:15:221
2890499139,3cyclictest29139-21sh23:10:010
2890499139,3cyclictest20617-21grep20:00:140
2890499139,3cyclictest16516-21tr21:25:190
2890499139,2cyclictest11775-21head19:40:160
2890499135,5cyclictest32540-21apache200:37:360
28904991312,1cyclictest0-21swapper/022:45:430
28904991310,2cyclictest9971-21grep23:55:270
28904991310,2cyclictest8915-21grep21:40:180
28904991310,2cyclictest8729-21latency_hist22:00:000
28904991310,2cyclictest3570-21grep22:35:130
28904991310,2cyclictest29928-21tr23:10:130
28904991310,2cyclictest27469-21grep22:30:140
28904991310,2cyclictest11541-21wc20:45:220
2890499130,7cyclictest17177-21idleruntime-cro20:59:590
2891299129,2cyclictest740-21cut20:25:123
2891299129,2cyclictest28374-21ssh23:47:323
2891299129,2cyclictest24782-21ssh21:30:213
2891299129,2cyclictest17856-21ssh00:00:233
2891299129,2cyclictest15748-21head20:55:173
2891299129,2cyclictest14883-21chrt00:37:043
2891299129,2cyclictest13794-21grep23:00:213
2891299129,2cyclictest13190-21basename19:45:013
2891299129,2cyclictest12084-21grep22:40:133
2891299128,3cyclictest1523-21sed21:35:283
2891299128,2cyclictest21512-21cut23:05:133
2891299127,2cyclictest452-21systemd-logind23:32:043
2891299127,2cyclictest225-21systemd-journal22:12:043
2891299121,5cyclictest0-21swapper/321:28:543
28912991210,2cyclictest39-21ksoftirqd/321:24:373
28912991210,2cyclictest19636-21ssh23:23:113
2891299120,5cyclictest0-21swapper/320:30:143
2891299120,4cyclictest0-21swapper/322:34:553
2891299120,11cyclictest1713-21grep00:10:163
2891099129,2cyclictest2596-21sh20:30:002
2891099129,2cyclictest23064-21tr20:05:142
2891099129,2cyclictest21842-21grep20:00:222
2891099129,2cyclictest14450-21cut23:20:162
2891099129,2cyclictest1193-21cron19:20:002
2891099129,2cyclictest1193-21cron19:19:592
2891099129,2cyclictest1186-21tr00:10:132
2891099125,5cyclictest0-21swapper/220:10:132
2890799129,2cyclictest7611-21cut23:35:171
2890799129,2cyclictest4590-21grep19:25:161
2890799129,2cyclictest28632-21grep20:15:141
2890799129,2cyclictest2214-21tail19:20:171
2890799129,2cyclictest21119-21find21:05:221
2890799129,2cyclictest20348-21wc20:00:001
2890799129,2cyclictest17440-21sed19:50:271
2890799129,2cyclictest15554-21ls19:50:011
2890799129,2cyclictest10419-21cut20:45:141
2890799128,3cyclictest10020-21ls20:44:591
2890799123,5cyclictest225-21systemd-journal22:17:001
28907991210,1cyclictest6294-21latency_hist23:35:001
28907991210,1cyclictest55650irq/127-enp2s0-22:03:211
2890799120,1cyclictest0-21swapper/122:54:171
2890799120,12cyclictest0-21swapper/121:49:471
2890499129,2cyclictest9526-21grep20:40:240
2890499129,2cyclictest5146-21taskset19:25:210
2890499129,2cyclictest4306-21grep00:30:270
2890499129,2cyclictest31725-21cut21:15:170
2890499129,2cyclictest27141-21sort00:25:210
2890499129,2cyclictest23872-21ls23:25:210
2890499129,2cyclictest19029-21grep21:00:270
2890499129,2cyclictest18415-21grep19:55:140
2890499129,2cyclictest14398-21grep23:20:160
2890499129,2cyclictest13835-21ssh23:00:220
2890499129,2cyclictest11048-21ssh00:15:260
2890499128,3cyclictest31924-21ssh00:09:240
28904991210,1cyclictest0-21swapper/022:20:120
2890499120,12cyclictest0-21swapper/021:23:340
2891299118,2cyclictest9728-21cat23:55:263
2891299118,2cyclictest8986-21ssh22:00:013
2891299118,2cyclictest6656-21ssh23:35:053
2891299118,2cyclictest6550-21sed20:35:203
2891299118,2cyclictest6341-21grep19:30:143
2891299118,2cyclictest5381-21sh23:15:033
2891299118,2cyclictest5256-21sh22:16:263
2891299118,2cyclictest4526-21aten2_r7power_c22:55:133
2891299118,2cyclictest4526-21aten2_r7power_c22:55:133
2891299118,2cyclictest3601-21apt19:25:013
2891299118,2cyclictest30492-21head23:10:153
2891299118,2cyclictest29102-21sh21:52:133
2891299118,2cyclictest28987-21ssh00:07:123
2891299118,2cyclictest25581-21tr20:10:143
2891299118,2cyclictest23645-21grep21:10:183
2891299118,2cyclictest21400-21ssh00:21:333
2891299118,2cyclictest21400-21ssh00:21:333
2891299118,2cyclictest16043-21ssh22:23:153
2891299118,2cyclictest10387-21grep19:35:263
2891299117,2cyclictest8147-21tr20:40:143
2891299117,2cyclictest12045-21sshd20:45:453
2891299115,1cyclictest20587-21apt-get20:00:133
2891099118,2cyclictest9762-21tail19:35:192
2891099118,2cyclictest9268-21ssh00:15:112
2891099118,2cyclictest8037-21cut20:40:122
2891099118,2cyclictest6623-21ssh21:20:052
2891099118,2cyclictest3260-21chrt19:20:472
2891099118,2cyclictest32399-21mailstats23:30:232
2891099118,2cyclictest31462-21ssh22:13:072
2891099118,2cyclictest28828-21grep22:50:152
2891099118,2cyclictest26349-21cut21:50:222
2891099118,2cyclictest23431-21ssh21:30:052
2891099118,2cyclictest23225-21head21:10:162
2891099118,2cyclictest21436-21ssh21:47:472
2891099118,2cyclictest20131-21ntpq22:25:212
2891099118,2cyclictest20084-21ssh21:27:392
2891099118,2cyclictest16774-21taskset20:55:312
2891099118,2cyclictest1521-21grep23:50:282
2891099117,3cyclictest6174-21head20:35:172
2891099116,2cyclictest32-21ksoftirqd/222:22:522
2891099110,1cyclictest0-21swapper/219:25:202
2891099110,11cyclictest0-21swapper/222:31:532
2891099110,10cyclictest4858-21tr22:55:162
2891099110,10cyclictest4858-21tr22:55:162
2891099110,10cyclictest14970-21expr23:00:312
2890799119,1cyclictest25-21ksoftirqd/122:59:281
2890799119,1cyclictest25-21ksoftirqd/122:59:271
2890799118,2cyclictest3299-21ssh22:35:111
2890799118,2cyclictest32447-21perf20:25:001
2890799118,2cyclictest2845-21ssh21:55:281
2890799118,2cyclictest24532-21wc20:05:261
2890799118,2cyclictest24126-21ssh23:25:241
2890799118,2cyclictest22191-21grep20:00:271
2890799118,2cyclictest16862-21ssh23:01:531
28907991110,1cyclictest0-21swapper/121:13:491
2890499118,2cyclictest7060-21egrep23:35:150
2890499118,2cyclictest6931-21tune2fs19:30:150
2890499118,2cyclictest4474-21ssh22:55:110
2890499118,2cyclictest4474-21ssh22:55:100
2890499118,2cyclictest32139-21grep19:15:170
2890499118,2cyclictest32139-21grep19:15:170
2890499118,2cyclictest25658-21expr20:10:140
2890499118,2cyclictest23602-21chrt23:45:110
2890499118,2cyclictest2114-21ssh23:50:490
2890499118,2cyclictest19126-21ssh21:46:000
2890499118,2cyclictest1752-21tr19:20:150
2890499118,2cyclictest15195-21ssh22:41:310
2890499111,4cyclictest18158-21wget00:20:150
2890499111,4cyclictest18158-21wget00:20:140
2890499110,10cyclictest5562-21ssh23:15:100
2891299108,1cyclictest38302sleep319:25:073
2891299108,1cyclictest32170-21chrt20:21:463
2891299108,1cyclictest14420-21latency19:45:193
2891299107,2cyclictest31661-21egrep19:15:153
2891299107,2cyclictest31661-21egrep19:15:143
2891099108,1cyclictest29506-21cat20:15:232
2891099108,1cyclictest19849-21cut21:05:152
2891099108,1cyclictest19017-21basename19:55:172
2891099108,1cyclictest13850-21expr19:45:152
2891099107,2cyclictest7037-21expr19:30:162
2891099107,2cyclictest3306-21ssh22:15:212
2891099107,2cyclictest30818-21grep20:20:162
2891099105,2cyclictest225-21systemd-journal19:45:002
2890799108,1cyclictest8338-21chrt19:34:151
2890799108,1cyclictest17428-21seq21:00:141
2890799107,2cyclictest9404-21grep19:35:171
2890799107,2cyclictest3879-21hostname20:30:181
2890799107,2cyclictest12797-21egrep20:50:141
2890799107,2cyclictest11834-21grep19:40:161
2890499108,1cyclictest5729-21ssh22:16:490
2890499108,1cyclictest24120-21sh20:05:200
2890499108,1cyclictest10598-21chrt19:37:470
2890499107,2cyclictest4787-21grep20:30:260
289129997,1cyclictest174192sleep321:00:133
289079996,2cyclictest29409-21tr19:10:151
289049997,1cyclictest28247-21expr20:15:130
289049996,2cyclictest420-21cat20:25:100
289049996,2cyclictest17131-21find19:50:230
289049996,2cyclictest13809-21ntpq20:50:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional