You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-16 - 14:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun Aug 16, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
363429364,9sleep30-21swapper/319:05:363
366029267,9sleep20-21swapper/219:05:562
388728975,9sleep00-21swapper/019:08:510
373328466,14sleep10-21swapper/119:06:511
317412630,0sleep00-21swapper/023:45:180
406999310,27cyclictest0-21swapper/222:30:122
406999280,24cyclictest0-21swapper/222:51:102
4074992724,2cyclictest39-21ksoftirqd/321:20:003
4074992621,1cyclictest39-21ksoftirqd/300:03:073
4074992519,2cyclictest39-21ksoftirqd/323:11:033
4074992420,1cyclictest39-21ksoftirqd/322:53:363
4074992419,1cyclictest39-21ksoftirqd/322:02:383
406999240,24cyclictest0-21swapper/221:12:352
4074992320,1cyclictest0-21swapper/320:29:543
406999231,21cyclictest0-21swapper/221:30:282
4074992219,1cyclictest0-21swapper/320:42:203
407499220,1cyclictest141rcu_preempt23:28:463
4069992222,0cyclictest0-21swapper/222:49:032
406999220,22cyclictest0-21swapper/223:21:422
406999220,21cyclictest0-21swapper/221:24:112
4066992221,1cyclictest0-21swapper/121:55:571
4074992119,0cyclictest0-21swapper/320:30:363
4074992118,1cyclictest0-21swapper/320:46:133
4074992118,1cyclictest0-21swapper/320:35:363
406999210,20cyclictest0-21swapper/223:09:242
406499210,21cyclictest0-21swapper/021:28:530
4074992018,2cyclictest31809-21kworker/3:121:56:463
4069992019,1cyclictest0-21swapper/223:26:122
406999200,19cyclictest0-21swapper/221:55:552
406699201,2cyclictest31288-21ps21:10:231
406499201,18cyclictest0-21swapper/019:55:140
406499192,2cyclictest0-21swapper/021:15:140
407499181,5cyclictest4724-21apt-config21:33:563
4074991812,2cyclictest39-21ksoftirqd/321:20:533
406999181,12cyclictest0-21swapper/222:35:292
406999180,13cyclictest11551-21ssh22:55:152
4066991816,2cyclictest29293-21diskmemload00:25:591
4074991715,2cyclictest0-21swapper/320:50:103
407499170,17cyclictest0-21swapper/300:12:283
4066991714,2cyclictest0-21swapper/100:33:121
406699170,16cyclictest0-21swapper/119:25:131
406499170,1cyclictest0-21swapper/000:33:420
4074991614,1cyclictest39-21ksoftirqd/322:57:003
4074991614,1cyclictest0-21swapper/323:04:073
4074991614,1cyclictest0-21swapper/323:04:073
4074991612,3cyclictest39-21ksoftirqd/321:45:013
4074991612,1cyclictest39-21ksoftirqd/323:42:363
4074991610,5cyclictest39-21ksoftirqd/322:25:013
4069991614,2cyclictest0-21swapper/200:09:332
406999160,16cyclictest0-21swapper/200:11:422
406999160,15cyclictest0-21swapper/222:22:502
406999160,12cyclictest0-21swapper/221:27:512
406699164,10cyclictest0-21swapper/120:35:151
406699162,6cyclictest225-21systemd-journal19:45:001
406699160,4cyclictest0-21swapper/119:30:151
406499164,7cyclictest14584-21awk23:55:000
406499162,13cyclictest8246-21awk22:15:000
407499159,3cyclictest39-21ksoftirqd/323:17:583
407499157,3cyclictest39-21ksoftirqd/323:07:583
407499150,6cyclictest0-21swapper/321:50:193
407499150,1cyclictest0-21swapper/323:47:103
407499150,1cyclictest0-21swapper/322:17:373
406999150,15cyclictest0-21swapper/221:50:002
406999150,14cyclictest0-21swapper/223:35:352
406999150,0cyclictest0-21swapper/223:15:232
406999150,0cyclictest0-21swapper/200:02:382
4066991513,1cyclictest0-21swapper/123:24:211
406699150,15cyclictest0-21swapper/123:08:241
406499152,10cyclictest141rcu_preempt22:32:170
4064991511,3cyclictest23114-21head23:40:190
406499150,2cyclictest0-21swapper/021:40:120
4074991412,2cyclictest39-21ksoftirqd/322:07:493
407499141,1cyclictest141rcu_preempt21:49:053
4074991410,3cyclictest1053-21grep22:10:173
4074991410,2cyclictest363-21wc00:25:003
406999149,4cyclictest32262-21wc20:05:252
4069991410,3cyclictest7121-21grep21:15:222
4069991410,3cyclictest31699-21sh23:45:192
406999140,12cyclictest0-21swapper/223:02:452
406999140,12cyclictest0-21swapper/223:02:442
406699149,4cyclictest24321-21cron00:20:011
406699141,1cyclictest29293-21diskmemload21:50:331
4066991411,2cyclictest25756-21cron00:40:001
4066991411,2cyclictest24477-21ssh22:05:131
406699141,12cyclictest21855-21sshd20:51:431
406699140,14cyclictest0-21swapper/121:23:181
406699140,13cyclictest0-21swapper/122:02:111
406499146,7cyclictest10984-21munin-run22:55:010
406499145,5cyclictest15973-21sh00:15:000
406499144,5cyclictest6295-21apache219:15:140
406499142,7cyclictest13394-21awk23:35:000
4064991414,0cyclictest0-21swapper/022:40:220
4064991413,1cyclictest29293-21diskmemload21:13:300
4064991413,1cyclictest181irq_work/022:08:020
4064991410,3cyclictest18347-21cron19:40:000
406499140,1cyclictest0-21swapper/023:57:040
406499140,1cyclictest0-21swapper/021:58:150
407499139,3cyclictest17477-21grep19:35:223
407499139,3cyclictest11447-21grep00:30:283
407499133,7cyclictest0-21swapper/322:40:133
4074991311,2cyclictest39-21ksoftirqd/323:20:463
4074991311,2cyclictest39-21ksoftirqd/321:39:233
4074991310,2cyclictest2743-21grep22:30:183
4074991310,2cyclictest27089-21ssh22:45:143
4074991310,2cyclictest25261-21tail19:50:273
4074991310,2cyclictest11264-21cat19:25:003
407499130,12cyclictest26453-21grep21:00:283
406999139,3cyclictest24509-21idleruntime-cro21:00:002
406999139,2cyclictest9868-21tr00:30:162
406999139,2cyclictest14575-21head20:35:292
406999138,3cyclictest14666-21wc21:40:002
4069991310,2cyclictest8150-21grep20:25:182
4069991310,2cyclictest3808-21head20:15:172
4069991310,2cyclictest18882-21grep20:45:242
406999130,2cyclictest0-21swapper/222:07:562
406999130,12cyclictest6368-21grep23:30:182
406999130,12cyclictest27482-21ssh22:25:212
406699139,3cyclictest585-21grep22:10:151
406699139,3cyclictest28086-21latency_hist20:00:001
406699139,3cyclictest21763-21tail23:00:241
406699139,3cyclictest21763-21tail23:00:231
406699139,3cyclictest19542-21cut22:40:171
406699139,2cyclictest27193-21grep22:45:151
406699138,4cyclictest32396-21grep20:05:251
406699131,12cyclictest29293-21diskmemload22:39:341
4066991311,1cyclictest0-21swapper/100:05:161
4066991310,2cyclictest26139-21sh21:26:491
4066991310,2cyclictest1368-21apt22:30:001
406699130,12cyclictest24750-21tr00:00:231
406699130,12cyclictest0-21swapper/123:53:301
406499132,6cyclictest19404-21sendmail-msp23:00:000
406499131,8cyclictest4356-21wget23:10:150
4064991310,2cyclictest26872-21head00:20:280
4064991310,2cyclictest1006-21cut00:05:260
407499129,2cyclictest6836-21grep19:15:153
407499129,2cyclictest30773-21ssh21:10:193
407499129,2cyclictest23373-21cat20:55:173
407499129,2cyclictest16483-21head23:55:223
407499129,2cyclictest16009-21cat19:35:003
407499125,1cyclictest39-21ksoftirqd/323:30:173
407499123,2cyclictest0-21swapper/300:15:233
407499122,3cyclictest141rcu_preempt22:29:323
407499120,0cyclictest0-21swapper/323:50:313
406999129,2cyclictest5917-21sed19:10:272
406999129,2cyclictest5088-21grep23:10:182
406999129,2cyclictest29019-21cut20:00:182
406999129,2cyclictest15287-21tr21:40:142
406999129,2cyclictest11607-21grep20:30:202
406999128,3cyclictest16007-21tail20:40:162
406999128,2cyclictest19374-21tail19:40:182
406999128,2cyclictest12203-21grep19:25:162
406999121,5cyclictest0-21swapper/223:50:142
4069991210,1cyclictest21306-21ssh22:03:022
4069991210,1cyclictest20912-21ssh00:17:192
406999120,3cyclictest0-21swapper/223:42:122
406999120,12cyclictest0-21swapper/221:51:412
406699129,2cyclictest3863-21grep20:15:171
406699129,2cyclictest31143-21sh21:50:001
406699129,2cyclictest30031-21grep23:25:151
406699129,2cyclictest2322-21grep20:10:271
406699129,2cyclictest18174-21tail22:20:181
406699129,2cyclictest11815-21grep22:55:161
406699129,2cyclictest10574-21grep20:30:141
406699129,2cyclictest10107-21apt-key20:30:001
406699128,3cyclictest7671-21chrt19:15:211
406699128,3cyclictest23312-21cat19:50:011
406699128,3cyclictest15862-21sed21:40:161
406699128,3cyclictest11373-21sh23:33:261
406699125,4cyclictest29293-21diskmemload21:30:121
406699125,3cyclictest225-21systemd-journal23:59:591
406699122,10cyclictest29293-21diskmemload22:16:471
406699121,10cyclictest29293-21diskmemload23:11:481
406499129,2cyclictest3727-21chrt00:26:060
406499129,2cyclictest32470-21sed20:05:250
406499129,2cyclictest29986-21grep20:00:270
406499129,2cyclictest22730-21grep19:45:280
406499129,2cyclictest13773-21sh22:36:370
406499129,2cyclictest12734-21grep19:25:210
406499121,6cyclictest7672-21awk20:25:000
406499120,7cyclictest29298-21perf21:10:010
406499120,6cyclictest14495-21sendmail21:40:000
406499120,11cyclictest31920-21ssh21:30:200
407499118,2cyclictest6456-21sh19:15:003
407499118,2cyclictest407-21munin-run20:10:003
407499118,2cyclictest3277-21ssh00:06:533
407499118,2cyclictest27450-21cut21:05:143
407499118,2cyclictest22885-21expr21:25:153
407499118,2cyclictest20835-21ssh23:39:183
407499118,2cyclictest18621-21cat19:40:133
407499118,2cyclictest13310-21chrt19:26:203
407499118,2cyclictest1108-21cut00:25:143
407499118,1cyclictest22054-21chrt00:37:073
407499111,5cyclictest225-21systemd-journal20:19:593
407499111,2cyclictest141rcu_preempt19:55:163
407499110,10cyclictest11936-21ssh22:35:253
406999118,2cyclictest9727-21grep19:20:162
406999118,2cyclictest8344-21grep19:15:282
406999118,2cyclictest29126-21chrt21:08:162
406999118,2cyclictest26728-21grep19:55:192
406999118,2cyclictest25389-21cut21:00:182
406999118,2cyclictest20870-21ssh23:58:252
406999118,2cyclictest19512-21ssh00:35:232
406999118,2cyclictest18690-21ssh22:40:122
406999118,2cyclictest1805-21grep20:10:192
406999118,2cyclictest1704-21rm00:25:172
406999118,2cyclictest12690-21ssh22:16:472
406999117,3cyclictest23740-21cut19:50:152
406999113,6cyclictest938-21ssh22:10:162
406999112,5cyclictest225-21systemd-journal00:25:002
406699118,2cyclictest9755-21uniq19:20:161
406699118,2cyclictest8911-21ssh21:35:371
406699118,2cyclictest7883-21ssh00:10:111
406699118,2cyclictest4400-21open_inodes22:50:211
406699118,2cyclictest31719-21cut23:45:181
406699118,2cyclictest29720-21chrt00:22:381
406699118,2cyclictest29476-21grep20:00:211
406699118,2cyclictest29007-21ssh23:44:071
406699118,2cyclictest23580-21grep20:55:201
406699118,2cyclictest1942-21grep22:30:131
406699118,2cyclictest19323-21ssh23:38:091
406699118,2cyclictest17527-21wc20:45:001
406699118,2cyclictest16386-21cut19:35:151
406699118,2cyclictest14701-21sed23:15:291
406699118,2cyclictest10608-21ssh21:17:431
406699117,3cyclictest19591-21chrt20:48:041
406499118,3cyclictest0-21swapper/022:02:130
406499118,2cyclictest332-21ssh23:26:270
406499118,2cyclictest30190-21grep23:05:270
406499118,2cyclictest29039-21sed22:45:280
406499118,2cyclictest26501-21sed21:00:270
406499118,2cyclictest26198-21chrt23:22:500
406499118,2cyclictest25790-21ssh23:03:280
406499118,2cyclictest25790-21ssh23:03:270
406499118,2cyclictest25458-21taskset19:51:500
406499118,2cyclictest25180-21grep21:45:250
406499118,2cyclictest23169-21grep20:55:150
406499118,2cyclictest22213-21taskset00:37:150
406499118,2cyclictest18448-21sed00:15:280
406499118,2cyclictest17116-21ssh21:21:150
406499118,2cyclictest16963-21grep20:40:250
406499118,2cyclictest14966-21ssh23:15:400
406499118,2cyclictest1458-21head20:10:180
406499118,2cyclictest13998-21ssh23:35:150
406499118,2cyclictest13449-21ssh22:17:220
406499117,3cyclictest23765-21tr00:00:160
406499115,5cyclictest13-21ksoftirqd/022:25:160
406499110,10cyclictest0-21swapper/022:21:210
407499100,9cyclictest29367-21ls20:00:203
406999108,2cyclictest6492-21grep20:20:182
406999108,1cyclictest20981-21chrt19:45:102
406999107,2cyclictest18120-21chrt19:37:092
406999106,3cyclictest14156-21grep19:30:172
406999102,3cyclictest225-21systemd-journal20:53:582
406699108,1cyclictest6342-21taskset19:13:381
406699108,1cyclictest24338-21tail19:50:171
406699105,4cyclictest27235-21tr21:05:131
406699100,9cyclictest265452sleep121:00:391
406499108,1cyclictest31498-21ssh21:50:060
406499108,1cyclictest21598-21sed20:50:250
406499108,1cyclictest18848-21cat19:40:150
406499108,1cyclictest14286-21expr19:30:170
406499108,1cyclictest10330-21dpkg20:30:000
406499107,2cyclictest5241-21sendmail-msp20:19:590
406499107,2cyclictest4497-21grep19:10:150
406499107,2cyclictest13123-21expr20:35:170
406499107,2cyclictest10108-21taskset19:20:200
40749997,1cyclictest1026-21expr20:10:143
40749996,2cyclictest6366-21basename20:20:173
40749992,3cyclictest225-21systemd-journal19:45:073
40669997,1cyclictest5818-21expr20:20:131
40649997,1cyclictest7893-21cat20:25:160
40649997,1cyclictest193912sleep020:45:300
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional