You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-07 - 05:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Aug 07, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1130529080,6sleep20-21swapper/219:08:312
1128929080,6sleep10-21swapper/119:08:181
1133528875,9sleep00-21swapper/019:08:550
1125628559,10sleep30-21swapper/319:07:523
147772810,0sleep10-21swapper/122:33:061
103192610,1sleep339-21ksoftirqd/323:28:213
1154099310,28cyclictest0-21swapper/119:10:251
1154099300,29cyclictest3500-21ssh23:05:131
11535992625,1cyclictest0-21swapper/022:00:110
1154599252,18cyclictest22800-21grep21:40:172
1154599251,1cyclictest141rcu_preempt22:04:332
11540992421,2cyclictest29701-21/usr/sbin/munin23:40:201
1155099232,20cyclictest0-21swapper/300:25:173
11550992320,2cyclictest1142-21runrttasks21:47:303
11540992320,2cyclictest1142-21runrttasks22:52:511
11550992219,2cyclictest16877-21sshd22:53:393
11550992219,2cyclictest16165-21df_abs00:10:143
1155099220,1cyclictest0-21swapper/321:54:563
11545992221,1cyclictest0-21swapper/200:27:482
11540992220,1cyclictest15941-21timerwakeupswit21:35:261
1154099221,2cyclictest504-21/usr/sbin/munin00:20:171
1153599222,2cyclictest141rcu_preempt22:51:410
1153599220,17cyclictest0-21swapper/022:11:230
31242210,0sleep20-21swapper/223:24:172
11550992119,1cyclictest1142-21runrttasks23:13:553
11550992118,2cyclictest16794-21ntp_states21:55:213
11550992118,2cyclictest1142-21runrttasks23:38:113
11550992118,2cyclictest1142-21runrttasks21:14:103
1155099210,17cyclictest0-21swapper/320:50:113
1154599210,19cyclictest0-21swapper/221:31:522
11540992111,9cyclictest0-21swapper/119:30:161
11550992018,1cyclictest20203-21/usr/sbin/munin23:15:253
11550992017,2cyclictest9433-21aten2_r7power_e22:30:143
1154599201,2cyclictest0-21swapper/221:53:272
11540992017,2cyclictest1142-21runrttasks23:58:251
11540992014,3cyclictest0-21swapper/123:51:551
1153599206,8cyclictest4093-21diskmemload22:00:000
1153599200,2cyclictest0-21swapper/000:27:480
11550991917,1cyclictest1142-21runrttasks21:17:003
11550991916,2cyclictest22461-21/usr/sbin/munin21:40:203
1154599190,2cyclictest0-21swapper/222:29:022
1154099196,7cyclictest225-21systemd-journal22:20:011
11540991917,1cyclictest5886-21cpu23:45:151
11540991917,1cyclictest21822-21cstates21:20:181
1153599199,4cyclictest0-21swapper/021:28:040
11550991816,1cyclictest7561-21df_abs00:05:163
11550991816,1cyclictest22949-21df23:55:163
1155099180,2cyclictest0-21swapper/300:37:233
1154599180,17cyclictest0-21swapper/221:20:102
1154099183,15cyclictest25-21ksoftirqd/100:35:471
11540991816,1cyclictest0-21swapper/100:25:231
1154099180,16cyclictest0-21swapper/122:07:531
11535991817,1cyclictest0-21swapper/022:43:210
11535991817,1cyclictest0-21swapper/000:09:400
1153599181,5cyclictest141rcu_preempt22:05:320
1153599180,1cyclictest0-21swapper/021:40:410
11550991715,1cyclictest1142-21runrttasks00:18:463
1155099170,1cyclictest31736-21unixbench_singl21:25:283
1154599173,6cyclictest19576-21apt-get20:35:032
1154599170,6cyclictest0-21swapper/200:35:332
1154599170,17cyclictest0-21swapper/222:05:252
1154599170,16cyclictest0-21swapper/221:39:532
11540991717,0cyclictest29251-21kworker/1:021:11:111
11540991715,1cyclictest28606-21/usr/sbin/munin23:20:201
1154099170,2cyclictest0-21swapper/123:00:151
1154099170,16cyclictest0-21swapper/100:11:111
1153599170,1cyclictest0-21swapper/021:39:580
128232160,0sleep10-21swapper/100:08:141
11550991613,2cyclictest30951-21aten2_r7power_e00:00:143
11550991613,2cyclictest1142-21runrttasks23:43:163
11550991613,2cyclictest1142-21runrttasks23:21:403
1155099161,14cyclictest0-21swapper/323:30:073
11550991611,3cyclictest15628-21wc19:15:263
11550991610,2cyclictest29398-21sort21:24:593
11545991613,3cyclictest32-21ksoftirqd/200:05:002
11545991611,3cyclictest30638-21sh21:45:012
1154599160,15cyclictest0-21swapper/223:12:032
1154599160,0cyclictest0-21swapper/222:11:232
11540991613,2cyclictest1142-21runrttasks21:42:451
1153599161,5cyclictest141rcu_preempt23:01:510
11550991513,1cyclictest27469-21/usr/sbin/munin23:00:193
11550991512,2cyclictest1142-21runrttasks22:11:493
1155099151,1cyclictest0-21swapper/300:30:383
1155099150,2cyclictest0-21swapper/322:01:083
11545991510,3cyclictest0-21swapper/222:52:022
1154599150,6cyclictest0-21swapper/223:34:512
1154599150,15cyclictest0-21swapper/221:29:452
1154599150,14cyclictest0-21swapper/222:48:492
11540991513,1cyclictest1142-21runrttasks22:27:501
11540991512,2cyclictest30886-21/usr/sbin/munin21:45:131
1154099150,14cyclictest0-21swapper/121:05:121
1153599159,6cyclictest13-21ksoftirqd/021:16:130
11535991513,1cyclictest0-21swapper/023:49:240
1153599150,9cyclictest0-21swapper/023:32:200
1153599150,15cyclictest0-21swapper/022:48:490
11550991412,1cyclictest1142-21runrttasks21:38:073
11550991410,3cyclictest2495-21grep22:45:173
1154599147,2cyclictest22297-21interrupts23:35:172
1154599140,14cyclictest0-21swapper/223:07:032
11540991412,1cyclictest13695-21fschecks_time21:15:161
11540991412,1cyclictest1142-21runrttasks00:34:121
11540991410,3cyclictest19673-21cron23:15:001
1154099140,12cyclictest0-21swapper/122:44:361
1153599145,8cyclictest20727-21sensors22:55:250
1153599141,7cyclictest2927-21awk19:59:590
1153599141,6cyclictest225-21systemd-journal00:37:400
11535991413,1cyclictest0-21swapper/023:22:240
11535991413,1cyclictest0-21swapper/000:20:030
11535991412,2cyclictest0-21swapper/000:17:130
11535991410,3cyclictest13-21ksoftirqd/021:52:410
1153599140,1cyclictest0-21swapper/023:57:480
1155099138,3cyclictest3107-21cat20:00:013
11550991311,1cyclictest16537-21/usr/sbin/munin22:15:193
11550991310,2cyclictest586-21wc19:54:593
11550991310,2cyclictest32747-21grep21:00:173
11550991310,2cyclictest24836-21tr19:35:203
11550991310,2cyclictest16584-21grep19:20:153
1154599136,6cyclictest32-21ksoftirqd/223:15:192
11545991311,1cyclictest420-21kworker/2:223:49:242
1154599130,9cyclictest0-21swapper/223:01:522
1154599130,12cyclictest0-21swapper/200:30:212
1154099139,3cyclictest25140-21grep20:45:141
1154099139,3cyclictest16211-21cut21:55:171
1154099139,2cyclictest30072-21grep21:25:161
11540991312,1cyclictest0-21swapper/122:00:191
1154099131,1cyclictest10720-21ssh21:32:451
11540991311,1cyclictest489-21cron23:19:591
1154099131,11cyclictest4093-21diskmemload00:19:061
11540991310,2cyclictest31103-21grep19:50:151
11540991310,2cyclictest1706-21cron21:05:011
1154099130,5cyclictest0-21swapper/122:13:151
1153599138,4cyclictest13-21ksoftirqd/000:12:570
1153599134,5cyclictest225-21systemd-journal23:16:400
1153599132,9cyclictest0-21swapper/021:10:540
1153599132,6cyclictest1142-21runrttasks19:40:250
1153599130,12cyclictest29243-21cut19:45:160
300832120,2chrt32-21ksoftirqd/200:18:412
1155099129,2cyclictest788-21wc00:20:193
1155099129,2cyclictest31297-21grep20:55:243
1155099129,2cyclictest25152-21grep20:45:153
1155099129,2cyclictest21353-21aten2_r7power_p19:30:143
1155099129,2cyclictest21029-21sed22:55:273
1155099129,2cyclictest14321-21grep20:20:253
1155099128,3cyclictest10153-21wc20:15:003
1155099128,2cyclictest26332-21grep19:40:143
11550991210,1cyclictest13622-21sh21:34:593
1155099120,1cyclictest0-21swapper/323:08:253
1154599129,2cyclictest9644-21tr22:30:152
1154599129,2cyclictest7823-21cat20:10:012
1154599129,2cyclictest7720-21grep00:05:172
1154599129,2cyclictest29676-21basename20:55:122
1154599129,2cyclictest27158-21ssh22:20:322
1154599129,2cyclictest27158-21ssh22:20:312
1154599129,2cyclictest23472-21cat22:00:002
1154599129,2cyclictest21382-21ssh23:54:362
1154599129,2cyclictest20549-21chrt00:12:452
1154599129,2cyclictest10904-21sh23:28:482
1154599128,4cyclictest24895-21cut19:35:212
1154599126,5cyclictest13201-21cat21:15:132
11545991211,1cyclictest0-21swapper/222:19:592
1154599120,5cyclictest0-21swapper/219:10:132
1154599120,3cyclictest0-21swapper/220:15:132
1154599120,1cyclictest141rcu_preempt19:30:152
1154599120,12cyclictest0-21swapper/222:40:152
1154599120,11cyclictest0-21swapper/222:55:142
1154099129,2cyclictest921-21cut19:55:141
1154099129,2cyclictest31691-21tail00:00:171
1154099129,2cyclictest30885-21wc20:55:201
1154099129,2cyclictest28021-21sed20:50:151
1154099129,2cyclictest23662-21grep20:40:201
1154099129,2cyclictest19359-21tr22:55:151
1154099128,2cyclictest16436-21wc19:20:001
1153599129,2cyclictest5394-21tail23:25:160
1153599129,2cyclictest3667-21ssh22:25:580
1153599129,2cyclictest28340-21cut20:50:180
1153599129,2cyclictest24936-21chrt23:36:410
1153599129,2cyclictest0-21swapper/023:07:520
1153599127,4cyclictest13-21ksoftirqd/020:55:110
1153599127,1cyclictest13-21ksoftirqd/023:52:270
1153599125,5cyclictest5732-21apt-get21:30:140
1153599124,7cyclictest16928-21apache_accesses00:30:130
1155099119,1cyclictest15762-21cut20:25:153
1155099118,2cyclictest7664-21sed23:45:263
1155099118,2cyclictest592-21sort22:05:163
1155099118,2cyclictest30166-21ssh22:41:583
1155099118,2cyclictest25915-21tail22:20:183
1155099118,2cyclictest25915-21tail22:20:173
1155099118,2cyclictest2229-21cat22:25:203
1155099118,2cyclictest18510-21chrt22:35:163
1155099118,2cyclictest15004-21ssh23:50:193
1155099118,2cyclictest11347-21cat20:15:193
1155099111,9cyclictest20736-21chrt19:26:473
1154599119,2cyclictest8826-21sh21:12:112
1154599119,2cyclictest2836-21taskset19:59:202
1154599119,2cyclictest22546-21hostname23:55:152
1154599119,1cyclictest3294-21tr20:00:162
1154599119,1cyclictest32763-21expr21:00:172
1154599119,1cyclictest28967-21grep20:50:242
1154599119,1cyclictest27749-21sed19:40:262
1154599119,1cyclictest19662-21sed22:35:262
1154599119,1cyclictest17701-21tr20:30:152
1154599119,1cyclictest17701-21tr20:30:142
1154599119,1cyclictest16530-21cat20:25:242
1154599118,3cyclictest29769-21apt-get23:40:142
1154599118,2cyclictest8251-21ls20:10:142
1154599118,2cyclictest32673-21cat00:20:162
1154599118,2cyclictest32510-21tail19:50:262
1154599118,2cyclictest30395-21taskset19:47:502
1154599110,2cyclictest0-21swapper/219:25:142
1154599110,10cyclictest3115-21ntpq21:05:192
1154599110,10cyclictest24878-21apache_accesses20:45:132
1154099118,3cyclictest38150irq/131-snd_hda19:35:221
1154099118,2cyclictest7197-21chrt23:25:551
1154099118,2cyclictest5153-21chrt20:03:201
1154099118,2cyclictest4942-21ssh22:46:041
1154099118,2cyclictest28196-21sh19:44:591
1154099118,2cyclictest21224-21sh23:35:101
1154099118,2cyclictest19992-21grep19:25:211
1154099118,2cyclictest19359-21taskset20:32:241
1154099118,2cyclictest19359-21taskset20:32:231
1154099118,2cyclictest18265-21expr22:35:151
1154099118,2cyclictest17651-21ntpq19:20:201
1154099118,2cyclictest15713-21ssh23:31:021
1154099117,3cyclictest28390-21ssh22:21:301
1154099117,3cyclictest28390-21ssh22:21:291
1154099117,2cyclictest12980-21tr20:20:141
1154099114,5cyclictest0-21swapper/121:50:161
1153599119,2cyclictest23754-21grep20:40:210
1153599119,2cyclictest23366-21sh22:38:160
1153599119,2cyclictest22530-21sh22:18:300
1153599119,2cyclictest1969-21sh00:01:390
1153599119,1cyclictest9381-21chrt20:10:320
1153599119,1cyclictest5755-21awk20:05:150
1153599119,1cyclictest5292-21cron20:05:000
1153599119,1cyclictest25944-21cat20:45:200
1153599119,1cyclictest212242sleep019:30:040
1153599118,2cyclictest30664-21sh21:45:020
1153599118,2cyclictest23165-21wc21:20:300
1153599118,2cyclictest2084-21cut21:05:140
1153599118,2cyclictest14843-21grep19:15:180
1153599118,2cyclictest12965-21head19:10:200
1153599118,2cyclictest11806-21ssh23:10:120
1153599118,2cyclictest11016-21tail20:15:160
1153599116,5cyclictest24372-21apache223:41:080
1153599112,9cyclictest0-21swapper/022:20:470
1153599112,9cyclictest0-21swapper/022:20:460
1153599110,5cyclictest0-21swapper/021:00:220
1153599110,10cyclictest18088-21cut20:30:170
1153599110,10cyclictest18088-21cut20:30:170
1155099108,1cyclictest7048-21expr20:05:283
1155099108,1cyclictest3425-21ls21:05:243
1155099107,2cyclictest28753-21grep19:45:143
1155099107,2cyclictest13556-21taskset19:11:273
1155099102,4cyclictest20007-21apt-get20:35:033
1154599109,1cyclictest17435-21seq19:20:202
1154599109,1cyclictest14572-21expr19:15:152
1154599109,1cyclictest12862-21grep20:20:142
1154599108,2cyclictest22407-21taskset20:40:042
1154099108,1cyclictest98942sleep120:13:141
1154099108,1cyclictest22051-21taskset20:38:451
1154099108,1cyclictest11350-21sed20:15:181
1154099107,2cyclictest15224-21grep20:25:121
1154099107,1cyclictest0-21swapper/120:05:191
1154099100,9cyclictest302682sleep119:46:131
1153599108,2cyclictest19149-21grep19:25:150
1153599108,2cyclictest16218-21ntpq20:25:180
1153599108,1cyclictest31129-21tr19:50:150
1153599108,1cyclictest21753-21grep20:35:280
1153599108,1cyclictest13824-21cut20:20:200
1153599107,2cyclictest9574-21ssh22:30:140
1153599100,3cyclictest225-21systemd-journal19:40:000
115509997,1cyclictest3178-21cat20:00:123
115509997,1cyclictest23370-21uname20:40:173
115509997,1cyclictest18103-21grep20:30:173
115509997,1cyclictest18103-21grep20:30:163
115359995,3cyclictest17085-21sed19:20:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional