You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-16 - 10:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sat May 16, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
680729581,9sleep00-21swapper/019:05:210
584329368,9sleep30-21swapper/319:05:053
765529280,8sleep20-21swapper/219:05:482
797528465,15sleep10-21swapper/119:09:571
22712750,0sleep20-21swapper/221:30:042
80792730,0sleep20-21swapper/200:06:532
269292640,0sleep00-21swapper/022:03:320
807299280,27cyclictest0-21swapper/222:02:142
8076992726,0cyclictest0-21swapper/322:40:233
807299250,25cyclictest0-21swapper/200:04:272
807299250,24cyclictest0-21swapper/222:39:552
807699231,1cyclictest4464-21ssh21:11:183
8072992321,1cyclictest0-21swapper/221:58:462
807699220,1cyclictest0-21swapper/321:08:133
807299221,1cyclictest26845-21/usr/sbin/munin23:40:192
807299220,22cyclictest0-21swapper/221:18:252
807099222,1cyclictest851-21diskmemload22:57:161
807299210,20cyclictest0-21swapper/222:06:352
807699200,2cyclictest1593-21kworker/3:221:04:013
807699200,2cyclictest1593-21kworker/3:220:58:273
807299201,1cyclictest0-21swapper/223:35:042
807299200,1cyclictest0-21swapper/223:22:242
252072200,0sleep10-21swapper/123:58:131
8076991918,1cyclictest0-21swapper/300:14:103
807299190,1cyclictest0-21swapper/200:12:572
807299190,19cyclictest0-21swapper/200:16:052
806799190,19cyclictest0-21swapper/022:39:560
807299180,18cyclictest0-21swapper/222:29:542
807299180,18cyclictest0-21swapper/221:11:232
8070991817,1cyclictest0-21swapper/123:15:191
807099180,18cyclictest0-21swapper/122:18:521
806799180,2cyclictest0-21swapper/022:50:120
8076991715,2cyclictest39-21ksoftirqd/323:15:553
807299174,1cyclictest141rcu_preempt21:21:552
807099170,17cyclictest0-21swapper/121:11:411
8067991716,1cyclictest0-21swapper/022:17:050
8076991616,0cyclictest0-21swapper/300:23:203
807699161,5cyclictest225-21systemd-journal00:04:593
807699160,15cyclictest0-21swapper/321:51:203
807299166,9cyclictest10012-21ssh23:30:132
8072991615,1cyclictest0-21swapper/222:45:242
807299161,2cyclictest0-21swapper/221:47:472
8072991612,3cyclictest32-21ksoftirqd/221:36:292
807299161,11cyclictest13648-21/usr/sbin/munin22:15:262
8072991610,5cyclictest16701-21latency_hist23:15:012
807299160,1cyclictest0-21swapper/222:53:242
807299160,16cyclictest0-21swapper/223:00:252
807299160,0cyclictest0-21swapper/222:12:482
807099169,5cyclictest38150irq/131-snd_hda19:20:231
807099162,4cyclictest2639-21df_abs20:05:161
8070991615,1cyclictest0-21swapper/100:22:501
807099161,1cyclictest851-21diskmemload23:49:101
807099160,1cyclictest0-21swapper/100:10:191
807099160,16cyclictest0-21swapper/121:25:531
806799168,6cyclictest22953-21unixbench_singl00:15:260
218062160,0sleep20-21swapper/222:20:012
8076991513,1cyclictest18680-21cpu23:35:133
807699150,3cyclictest16023-21ssh21:56:323
807099154,10cyclictest30647-21ntp_states22:05:221
807099153,11cyclictest851-21diskmemload22:50:191
807099150,1cyclictest0-21swapper/123:29:121
8067991514,1cyclictest0-21swapper/023:21:590
33512151,1sleep1851-21diskmemload22:28:021
8076991413,1cyclictest0-21swapper/323:44:493
8076991412,1cyclictest8138-21apt-get23:10:033
8076991411,2cyclictest4803-21cron22:10:003
8076991410,3cyclictest923-21grep23:05:153
807299140,3cyclictest0-21swapper/223:45:402
807299140,13cyclictest0-21swapper/200:26:592
807299140,13cyclictest0-21swapper/200:22:222
807099149,3cyclictest12995-21wc19:20:001
8070991414,0cyclictest0-21swapper/100:18:101
8070991411,2cyclictest15896-21grep20:30:261
8070991410,3cyclictest38150irq/131-snd_hda23:35:051
8070991410,2cyclictest31158-21basename21:05:121
807099140,14cyclictest25800-21ssh22:21:471
806799149,4cyclictest13-21ksoftirqd/023:27:010
806799142,9cyclictest8267-21/usr/sbin/munin19:10:110
806799140,2cyclictest0-21swapper/022:25:150
807699139,2cyclictest2724-21cut20:05:173
807699136,2cyclictest29921-21awk22:24:593
807699133,9cyclictest0-21swapper/321:40:413
8076991313,0cyclictest0-21swapper/322:25:463
8076991310,2cyclictest22775-21rm22:40:003
8076991310,2cyclictest16548-21grep19:25:213
8076991310,2cyclictest16548-21grep19:25:203
8076991310,2cyclictest15381-21mailstats00:30:223
8076991310,2cyclictest11745-21wc20:25:003
807699130,1cyclictest141rcu_preempt21:38:433
807699130,1cyclictest0-21swapper/321:23:053
807699130,12cyclictest14038-21cat20:30:003
807299138,4cyclictest30999-21perf22:45:002
8072991312,1cyclictest32-21ksoftirqd/220:15:152
807299130,8cyclictest0-21swapper/221:52:522
807099139,4cyclictest38150irq/131-snd_hda20:40:171
807099139,3cyclictest10008-21grep19:10:271
807099139,2cyclictest18431-21grep20:35:251
807099138,3cyclictest0-21swapper/123:20:131
8070991312,1cyclictest0-21swapper/122:48:121
8070991311,1cyclictest851-21diskmemload22:32:461
8070991310,2cyclictest9626-21grep20:20:151
8070991310,2cyclictest22435-21wc19:40:001
8070991310,2cyclictest11682-21grep21:15:271
8070991310,2cyclictest0-21swapper/122:40:231
8070991310,1cyclictest8061-21cyclictest23:06:391
807099130,1cyclictest0-21swapper/123:52:351
807099130,13cyclictest0-21swapper/123:13:411
806799134,6cyclictest851-21diskmemload23:38:100
806799132,6cyclictest12668-21apache_accesses00:10:100
806799132,1cyclictest0-21swapper/021:53:010
8067991310,2cyclictest31872-21cron20:00:010
8067991310,2cyclictest30173-21head00:20:180
8067991310,2cyclictest29720-21uname00:00:240
8067991310,2cyclictest17663-21cat19:30:000
8067991310,2cyclictest17663-21cat19:30:000
8067991310,2cyclictest12206-21cron00:09:590
806799130,1cyclictest0-21swapper/023:05:590
806799130,12cyclictest9249-21basename21:15:000
806799130,12cyclictest11178-21wc21:15:230
807699129,2cyclictest32413-21grep20:00:143
807699129,2cyclictest30803-21grep19:55:203
807699129,2cyclictest3078-21ssh23:25:223
807699129,2cyclictest25213-21grep19:45:143
807699129,2cyclictest22396-21wc19:40:003
807699129,2cyclictest21656-21grep22:00:143
807699129,2cyclictest19469-21tr20:40:143
807699129,2cyclictest18055-21grep19:30:143
807699129,2cyclictest16179-21tr22:55:143
807699129,2cyclictest15626-21grep22:15:283
807699129,2cyclictest12347-21grep19:15:273
807699124,7cyclictest20951-21/usr/sbin/munin00:15:153
807699122,3cyclictest0-21swapper/321:45:133
8076991212,0cyclictest0-21swapper/321:31:233
807699120,12cyclictest0-21swapper/321:18:473
807699120,11cyclictest6171-21grep22:10:163
807299129,2cyclictest414-21head20:00:162
807299129,2cyclictest30448-21tail19:55:172
807299129,2cyclictest22808-21chrt23:18:152
807299129,2cyclictest1852-21tr23:25:152
807299129,2cyclictest17783-21sed22:55:252
807299129,2cyclictest11610-21tr23:50:152
807299129,2cyclictest1056-21grep23:05:152
807299128,3cyclictest11326-21ssh22:32:442
807299122,4cyclictest26964-21sshd19:48:432
807299121,5cyclictest19602-21/usr/sbin/munin21:40:122
807299120,3cyclictest0-21swapper/200:32:322
807299120,11cyclictest30935-21cat21:05:012
807099129,2cyclictest7703-21ssh00:06:301
807099129,2cyclictest7387-21grep00:25:261
807099129,2cyclictest4370-21tail21:30:241
807099129,2cyclictest31387-21tail19:55:271
807099129,2cyclictest29133-21head00:00:191
807099129,2cyclictest26069-21cron20:55:001
807099129,2cyclictest24537-21tr23:00:151
807099129,2cyclictest20467-21tail21:40:181
807099129,2cyclictest19128-21rm21:20:181
807099129,2cyclictest12490-21grep21:35:211
807099128,3cyclictest32238-21tr20:00:131
807099128,3cyclictest28258-21sed23:40:211
807099128,3cyclictest18878-21wc19:30:191
807099128,3cyclictest14598-21grep21:55:261
807099128,2cyclictest26608-21grep20:55:161
807099127,4cyclictest6131-21chrt21:50:311
807099126,5cyclictest22628-21perf22:40:001
807099125,4cyclictest851-21diskmemload21:50:001
807099124,5cyclictest851-21diskmemload00:30:241
807099120,2cyclictest141rcu_preempt00:36:571
806799129,2cyclictest7551-21ssh22:30:190
806799129,2cyclictest693-21cat20:00:180
806799129,2cyclictest6026-21grep22:10:160
806799129,2cyclictest30733-21grep22:05:210
806799129,2cyclictest29898-21sh22:25:000
806799129,2cyclictest21733-21cat19:35:230
806799129,2cyclictest16370-21ssh21:56:490
806799129,2cyclictest15990-21tr22:55:130
806799129,2cyclictest14481-21tr20:30:140
806799129,2cyclictest10235-21ssh23:30:150
806799127,4cyclictest3559-21basename21:50:000
806799127,4cyclictest24372-21cut23:00:140
8067991210,1cyclictest3049-21grep23:45:130
806799120,9cyclictest0-21swapper/022:45:520
806799120,1cyclictest0-21swapper/020:05:180
807699118,2cyclictest9611-21grep19:10:223
807699118,2cyclictest7010-21mailstats00:25:223
807699118,2cyclictest4392-21basename00:05:153
807699118,2cyclictest3758-21tail23:45:163
807699118,2cyclictest31600-21cut22:45:163
807699118,2cyclictest30860-21ssh21:27:183
807699118,2cyclictest27023-21sh23:01:023
807699118,2cyclictest26393-21tail23:20:163
807699118,2cyclictest25897-21ssh00:36:443
807699118,2cyclictest24221-21grep20:50:153
807699118,2cyclictest22712-21sh23:56:163
807699118,2cyclictest21810-21grep20:45:133
807699118,2cyclictest17411-21forks20:35:153
807699118,2cyclictest16334-21cron20:35:003
807699118,2cyclictest15044-21ssh22:54:463
807699118,2cyclictest12853-21ssh23:31:073
807699118,2cyclictest11172-21ssh23:50:093
807699118,2cyclictest10675-21ssh22:32:133
807299119,1cyclictest22390-21grep00:35:152
807299118,2cyclictest32141-21cat21:05:172
807299118,2cyclictest3129-21sed20:05:202
807299118,2cyclictest27459-21cut19:50:142
807299118,2cyclictest19393-21sh23:55:022
807299118,2cyclictest18390-21wc20:35:242
807299118,2cyclictest15712-21tr19:25:152
807299118,2cyclictest15712-21tr19:25:142
807299118,2cyclictest14736-21sed20:30:162
807299118,2cyclictest13092-21head20:25:212
807299117,3cyclictest27309-21sed21:25:172
807299114,4cyclictest225-21systemd-journal20:45:002
807299110,10cyclictest10998-21wc20:20:262
807099118,2cyclictest7012-21sort22:10:241
807099118,2cyclictest5044-21grep20:10:141
807099118,2cyclictest28048-21tail19:50:171
807099118,2cyclictest25891-21ssh22:02:431
807099118,2cyclictest25203-21tr19:45:141
807099118,2cyclictest13511-21tail20:25:251
807099118,2cyclictest12502-21ssh23:30:511
807099117,3cyclictest31032-21cat21:05:011
807099110,10cyclictest23289-21gltestperf19:40:161
806799119,2cyclictest9180-21ssh00:26:430
806799119,2cyclictest31268-21ssh23:42:170
806799119,2cyclictest25469-21sed19:45:150
806799119,2cyclictest14977-21cat00:30:170
806799119,2cyclictest10975-21ssh23:10:320
806799119,2cyclictest0-21swapper/022:40:420
806799119,1cyclictest7728-21grep20:15:160
806799119,1cyclictest4825-21expr20:10:130
806799119,1cyclictest25063-21ssh21:43:070
806799119,1cyclictest23068-21cat20:45:240
806799119,1cyclictest22324-21rm00:35:160
806799119,1cyclictest18941-21/usr/sbin/munin21:40:000
806799119,1cyclictest17854-21ls20:35:190
806799119,1cyclictest17151-21rm23:15:110
806799118,2cyclictest9903-21expr20:20:150
806799118,2cyclictest27502-21uname21:25:170
806799118,2cyclictest25610-21grep20:50:260
806799118,2cyclictest20486-21sed23:55:180
806799118,2cyclictest20302-21uname20:40:190
806799118,2cyclictest13058-21cat19:20:130
806799118,2cyclictest12586-21grep20:25:160
806799116,4cyclictest477-21rs:main0
806799110,4cyclictest0-21swapper/021:31:020
806799110,10cyclictest18013-21ssh21:20:080
807699108,2cyclictest29212-21taskset19:52:043
807699108,1cyclictest5806-21grep20:10:213
807699108,1cyclictest22733-21egrep19:40:133
807699107,2cyclictest7045-21seq20:15:053
807699107,2cyclictest13356-21grep19:20:153
807299108,1cyclictest8978-21grep19:10:162
807299108,1cyclictest196532sleep219:31:362
807299107,2cyclictest4959-21expr20:10:132
807299107,2cyclictest26423-21cut20:55:162
807299107,2cyclictest25740-21taskset20:50:482
807299107,2cyclictest24729-21wc19:45:012
807299107,2cyclictest23090-21sed20:45:252
807299107,2cyclictest20155-21chrt19:35:042
807299107,2cyclictest14504-21sed19:20:252
807299107,2cyclictest10868-21ls19:15:152
807099108,1cyclictest22000-21expr20:45:141
807099107,2cyclictest15413-21chrt19:25:091
807099107,2cyclictest15413-21chrt19:25:091
806799109,1cyclictest11532-21if_enp0s31f619:15:180
806799108,2cyclictest24415-21seq19:41:280
806799108,1cyclictest27297-21chrt19:50:010
806799108,1cyclictest26379-21cut20:55:160
806799107,2cyclictest28703-21chrt21:00:100
80709997,1cyclictest7991-21cat20:15:181
80679998,1cyclictest31467-21cat21:05:130
80679998,1cyclictest18224-21expr19:30:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional