You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-03 - 13:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Apr 03, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
108329986,9sleep00-21swapper/019:09:550
93928860,9sleep30-21swapper/319:08:053
3190528862,9sleep20-21swapper/219:05:152
3263628464,3sleep10-21swapper/119:05:221
317082610,0sleep10-21swapper/123:08:311
52622560,0sleep10-21swapper/122:14:301
147202550,0sleep10-21swapper/121:21:211
13782540,0sleep30-21swapper/322:30:363
119099290,14cyclictest0-21swapper/223:16:432
119099268,1cyclictest26605-21diskmemload23:40:542
119099262,9cyclictest13642-21runrttasks22:57:332
119099260,12cyclictest0-21swapper/223:32:272
119099260,10cyclictest0-21swapper/222:44:182
125812250,1sleep00-21swapper/022:37:460
118799250,1cyclictest0-21swapper/123:36:331
119199231,0cyclictest0-21swapper/323:58:523
119199230,0cyclictest0-21swapper/322:41:413
119099230,7cyclictest0-21swapper/223:46:412
119099230,23cyclictest0-21swapper/222:16:092
119099230,22cyclictest0-21swapper/223:57:402
119099230,17cyclictest0-21swapper/222:51:272
119099222,3cyclictest7033-21sh00:30:142
119099221,21cyclictest0-21swapper/223:08:422
119099220,10cyclictest0-21swapper/200:05:202
118799221,20cyclictest23066-21perf19:54:591
119199200,3cyclictest0-21swapper/321:15:253
119099207,0cyclictest0-21swapper/221:22:012
119099200,4cyclictest0-21swapper/221:32:292
119099200,19cyclictest0-21swapper/223:11:502
119099193,6cyclictest5947-21timerandwakeup23:50:252
1190991914,4cyclictest1436-21sendmail-mta21:54:392
119099190,19cyclictest0-21swapper/221:40:552
119099190,19cyclictest0-21swapper/221:16:532
119199183,1cyclictest141rcu_preempt23:30:073
1191991817,1cyclictest0-21swapper/323:01:293
1191991817,1cyclictest0-21swapper/322:27:493
1191991816,2cyclictest39-21ksoftirqd/300:31:193
119199180,0cyclictest0-21swapper/323:27:463
119099183,3cyclictest18011-21sh23:00:162
1190991816,1cyclictest0-21swapper/200:20:442
119099180,18cyclictest0-21swapper/200:39:192
119099180,17cyclictest0-21swapper/221:39:292
119099180,15cyclictest0-21swapper/223:35:052
119099180,15cyclictest0-21swapper/221:45:232
119099180,14cyclictest0-21swapper/222:30:062
119099180,0cyclictest0-21swapper/222:21:242
118799180,18cyclictest0-21swapper/123:33:081
118799180,18cyclictest0-21swapper/121:49:421
118599184,7cyclictest14351-21munin-run22:20:000
119099179,2cyclictest31668-21cat22:10:192
119099178,2cyclictest2587-21cron21:15:012
1190991710,2cyclictest25989-21cut20:00:122
119099170,8cyclictest225-21systemd-journal19:39:592
1187991716,1cyclictest0-21swapper/120:30:121
1187991715,2cyclictest26605-21diskmemload00:21:161
118799171,15cyclictest5996-21ssh00:10:131
118799170,16cyclictest0-21swapper/123:03:571
118799170,16cyclictest0-21swapper/121:40:551
118799170,0cyclictest0-21swapper/122:35:431
203092161,1sleep326605-21diskmemload22:04:003
119199160,3cyclictest0-21swapper/322:18:233
119199160,15cyclictest0-21swapper/300:10:273
119099168,2cyclictest19442-21taskset23:20:182
119099160,3cyclictest225-21systemd-journal19:55:232
119099160,13cyclictest0-21swapper/222:49:562
119099160,12cyclictest0-21swapper/223:28:132
119099160,0cyclictest0-21swapper/200:19:282
119099160,0cyclictest0-21swapper/200:19:272
1187991615,1cyclictest0-21swapper/123:42:231
118799161,2cyclictest141rcu_preempt21:16:351
118799160,16cyclictest0-21swapper/121:13:031
118599162,7cyclictest11066-21munin-run23:34:590
1185991615,1cyclictest0-21swapper/021:44:340
1185991613,2cyclictest1436-21sendmail-mta22:49:390
50872150,0chrt0-21swapper/022:33:290
1191991511,3cyclictest30428-21grep00:05:183
119099157,2cyclictest21042-21grep19:50:152
119099152,1cyclictest0-21swapper/219:20:112
1190991511,2cyclictest3733-21cron19:15:012
119099150,15cyclictest0-21swapper/200:26:412
119099150,11cyclictest6126-21latency_hist19:20:012
118799152,11cyclictest26212-21grep21:05:271
1187991514,1cyclictest0-21swapper/123:59:481
1187991513,2cyclictest0-21swapper/122:44:421
1187991513,1cyclictest0-21swapper/123:53:071
1187991512,3cyclictest26605-21diskmemload00:38:251
1187991512,2cyclictest25549-21mii-tool22:45:191
1187991512,2cyclictest1436-21sendmail-mta00:09:141
1187991511,4cyclictest8866-21apache221:39:391
118799150,14cyclictest0-21swapper/100:15:131
118799150,14cyclictest0-21swapper/100:15:131
118599157,7cyclictest27741-21munin-run21:30:000
1185991513,1cyclictest1436-21sendmail-mta23:52:540
118599150,3cyclictest0-21swapper/022:02:450
119199142,9cyclictest16329-21irqstats00:35:193
1191991410,3cyclictest31107-21tr00:25:153
1191991410,3cyclictest15672-21grep22:20:173
119199140,11cyclictest0-21swapper/323:21:353
1190991410,3cyclictest7400-21wc00:10:222
1190991410,3cyclictest5681-21wc21:55:132
118799141,3cyclictest0-21swapper/100:25:141
1185991412,2cyclictest13-21ksoftirqd/000:11:360
1185991412,1cyclictest1436-21sendmail-mta23:27:240
1185991411,2cyclictest1436-21sendmail-mta21:51:290
119199139,3cyclictest4991-21head23:50:163
119199139,3cyclictest24763-21grep21:05:163
119199139,3cyclictest15112-21grep20:45:133
119199139,3cyclictest12589-21ssh21:20:193
119199132,5cyclictest0-21swapper/322:55:133
119199131,12cyclictest0-21swapper/323:45:223
1191991310,2cyclictest4917-21grep21:35:163
1191991310,2cyclictest3301-21grep19:10:263
1191991310,2cyclictest30300-21cat20:10:003
1191991310,2cyclictest22463-21ntp_states21:45:213
119199130,1cyclictest0-21swapper/323:17:193
119099139,3cyclictest17371-21basename20:50:112
119099131,4cyclictest26848-21ssh21:29:192
1190991310,2cyclictest16909-21head19:40:172
119099130,2cyclictest0-21swapper/222:39:212
118799139,3cyclictest23005-21head00:00:271
118799139,3cyclictest10294-21ssh22:55:221
118799138,4cyclictest11398-21grep19:30:131
118799133,7cyclictest28681-21fschecks_count20:05:171
1187991312,1cyclictest25-21ksoftirqd/121:26:581
1187991310,2cyclictest27809-21cron20:05:001
1187991310,2cyclictest27730-21tune2fs23:25:171
1187991310,2cyclictest16796-21grep22:20:281
1187991310,2cyclictest1436-21sendmail-mta23:15:531
118799130,13cyclictest0-21swapper/122:29:081
118799130,12cyclictest16659-21sed20:45:251
118799130,12cyclictest0-21swapper/100:33:471
118599138,4cyclictest32561-21cron20:15:000
118599138,3cyclictest14703-21cron20:45:010
118599138,3cyclictest14703-21cron20:45:000
118599132,5cyclictest8537-21perf19:25:000
1185991310,2cyclictest25986-21grep20:00:130
1185991310,2cyclictest22449-21head00:00:210
1185991310,2cyclictest18024-21sort20:50:140
118599130,1cyclictest0-21swapper/021:45:120
118599130,13cyclictest0-21swapper/023:18:550
119199129,2cyclictest7167-21head19:20:153
119199129,2cyclictest6609-21sh21:55:183
119199129,2cyclictest5479-21tr20:25:143
119199129,2cyclictest30840-21grep21:50:233
119199129,2cyclictest28820-21cat21:10:273
119199129,2cyclictest27878-21cat21:30:003
119199129,2cyclictest24433-21grep19:55:203
119199129,2cyclictest15947-21basename19:39:593
119199128,3cyclictest20200-21tr23:40:143
119199128,2cyclictest21010-21tr19:50:143
119199126,5cyclictest8877-21cups-browsed19:25:223
119199122,3cyclictest225-21systemd-journal19:45:003
119199122,1cyclictest0-21swapper/300:22:033
119199120,1cyclictest0-21swapper/323:35:303
119199120,1cyclictest0-21swapper/320:55:123
119099129,2cyclictest23035-21ssh00:00:272
119099129,2cyclictest16587-21grep20:45:252
119099129,2cyclictest11652-21wc20:35:242
119099128,2cyclictest23533-21ls21:00:242
119099128,1cyclictest0-21swapper/222:01:592
119099127,4cyclictest30010-21ssh22:10:002
118799129,2cyclictest6173-21head20:25:181
118799129,2cyclictest5139-21idleruntime-cro20:25:001
118799129,2cyclictest2561-21grep23:10:171
118799127,4cyclictest18863-21ssh22:02:501
118799127,4cyclictest18666-21cut19:45:131
118799126,2cyclictest225-21systemd-journal20:40:001
118799120,1cyclictest0-21swapper/121:55:121
118599129,2cyclictest4129-21grep20:20:180
118599129,2cyclictest31622-21sed00:25:160
118599129,2cyclictest30520-21basename00:05:180
118599129,2cyclictest29984-21cat23:45:260
118599129,2cyclictest26140-21grep21:05:270
118599129,2cyclictest23816-21cat00:20:190
118599129,2cyclictest13417-21head23:55:160
118599129,2cyclictest13266-21latency_hist22:00:000
118599129,2cyclictest12352-21head21:20:160
118599128,3cyclictest15056-21wc19:35:200
118599124,5cyclictest18635-21apt-get23:20:120
1185991210,1cyclictest0-21swapper/022:27:590
118599120,0cyclictest0-21swapper/020:35:120
119199119,1cyclictest29431-21grep21:30:193
119199118,2cyclictest9061-21head20:30:213
119199118,2cyclictest4289-21ssh22:13:443
119199118,2cyclictest3387-21ls23:10:263
119199118,2cyclictest3253-21tr20:20:133
119199118,2cyclictest307-21chrt22:50:063
119199118,2cyclictest29671-21ssh22:47:523
119199118,2cyclictest27825-21awk20:04:593
119199118,2cyclictest24541-21ssh22:05:453
119199118,2cyclictest22604-21grep21:00:133
119199118,2cyclictest21088-21grep00:00:133
119199118,2cyclictest20666-21cron19:49:593
119199118,2cyclictest18099-21expr20:50:153
119199118,2cyclictest17242-21ssh00:16:123
119199118,2cyclictest17242-21ssh00:16:113
119199118,2cyclictest15103-21ssh21:40:383
119199118,2cyclictest1372-21tail20:15:153
119199118,2cyclictest13086-21chrt19:32:083
119199118,2cyclictest10279-21ssh22:35:573
119199117,3cyclictest25543-21ssh23:05:063
119199116,2cyclictest8866-21apache220:35:113
119199113,4cyclictest225-21systemd-journal19:19:563
119099118,2cyclictest8895-21cat20:30:192
119099118,2cyclictest3308-21tr20:20:132
119099118,2cyclictest32121-21grep20:10:262
119099118,2cyclictest30163-21taskset20:09:472
119099118,2cyclictest24027-21ssh22:25:172
119099118,2cyclictest14717-21munin-run20:45:012
119099118,2cyclictest14717-21munin-run20:45:002
119099118,2cyclictest13090-21chrt19:32:112
118799118,2cyclictest6678-21grep19:20:131
118799118,2cyclictest3736-21cron19:15:001
118799118,2cyclictest31630-21sh22:30:141
118799118,2cyclictest31225-21ssh23:46:101
118799118,2cyclictest29303-21ssh21:50:081
118799118,2cyclictest28494-21cat21:30:141
118799118,2cyclictest26227-21ssh23:24:491
118799118,2cyclictest17904-21sed19:40:261
118799118,2cyclictest17417-21cut20:50:111
118799118,2cyclictest15469-21tail19:35:241
118799118,2cyclictest1376-21if_enp0s31f622:50:141
118799117,3cyclictest0-21swapper/122:06:571
118799116,4cyclictest5233-21head19:15:211
118799116,4cyclictest2417-21taskset20:15:561
118799110,9cyclictest7616-21missed_timers22:15:211
118799110,10cyclictest23362-21cut19:55:131
118599118,2cyclictest8987-21egrep19:25:130
118599118,2cyclictest5285-21seq21:15:350
118599118,2cyclictest30397-21ssh22:10:100
118599118,2cyclictest29813-21ssh21:10:590
118599118,2cyclictest29776-21sed20:05:260
118599118,2cyclictest28035-21ssh21:30:030
118599118,2cyclictest26027-21cut23:05:130
118599118,2cyclictest25608-21ssh23:43:190
118599118,2cyclictest24934-21sed19:55:260
118599118,2cyclictest22542sleep023:10:160
118599118,2cyclictest22348-21tr21:00:130
118599118,2cyclictest21861-21ssh22:43:260
118599118,2cyclictest21856-21readlink21:00:010
118599118,2cyclictest20493-21ssh23:01:200
118599118,2cyclictest18899-21expr19:45:140
118599118,2cyclictest17547-21ssh00:35:380
118599118,2cyclictest15804-21systemctl00:15:220
118599118,2cyclictest15804-21systemctl00:15:220
118599118,2cyclictest1261-21tr22:50:130
118599118,2cyclictest11531-21basename23:35:120
118599118,2cyclictest10753-21sh00:31:500
118599114,5cyclictest1142-21runrttasks22:20:150
118599110,10cyclictest0-21swapper/021:35:130
119199108,1cyclictest31518-21ls20:10:203
119199108,1cyclictest13346-21tr20:40:173
119199108,1cyclictest13346-21tr20:40:163
119099108,1cyclictest26766-21cat21:10:002
119099108,1cyclictest106242sleep219:25:582
119099107,2cyclictest6714-21gzip20:25:232
119099107,2cyclictest577-21chrt20:15:062
119099107,2cyclictest18668-21wget19:45:132
118799107,2cyclictest9125-21tr19:25:141
118799107,2cyclictest32305-21taskset20:11:471
118799107,2cyclictest21481-21taskset20:55:401
118799100,9cyclictest12805-21cat20:40:141
118799100,9cyclictest12805-21cat20:40:131
118799100,2cyclictest225-21systemd-journal21:00:401
118599108,1cyclictest8104-21expr20:30:130
118599108,1cyclictest3996-21bc19:15:110
118599107,2cyclictest6159-21head20:25:180
118599107,2cyclictest16842-21grep19:40:160
118599107,2cyclictest10647-21ssh22:55:260
118599106,3cyclictest0-21swapper/019:30:200
118599100,9cyclictest22563-21expr22:05:150
118599100,6cyclictest225-21systemd-journal19:54:590
118599100,3cyclictest225-21systemd-journal20:20:000
11909996,2cyclictest19681-21taskset20:55:042
11859997,1cyclictest15118-21grep20:45:130
11859990,9cyclictest2781-21ntpq19:10:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional