You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-20 - 10:21

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed May 20, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
370429784,9sleep00-21swapper/019:08:380
368129582,9sleep20-21swapper/219:08:192
370028463,6sleep10-21swapper/119:08:341
179028368,10sleep30-21swapper/319:05:153
92562680,0sleep10-21swapper/122:34:431
112832620,0sleep00-21swapper/000:11:430
389299271,10cyclictest29261-21wget23:25:140
3896992424,0cyclictest0-21swapper/122:38:141
390399236,9cyclictest14033-21dump-pmu-power23:55:003
390199230,23cyclictest0-21swapper/222:49:542
389699230,2cyclictest0-21swapper/123:13:161
390199220,20cyclictest0-21swapper/221:18:202
3896992117,3cyclictest29728-21cat20:00:251
389699211,19cyclictest0-21swapper/120:20:131
389699210,21cyclictest0-21swapper/100:17:141
389699210,1cyclictest0-21swapper/121:14:011
390399200,19cyclictest0-21swapper/323:35:193
390199200,20cyclictest0-21swapper/222:16:552
390199200,20cyclictest0-21swapper/200:19:262
3896992019,1cyclictest0-21swapper/123:46:531
3896992017,2cyclictest13395-21perf19:30:001
389699200,1cyclictest0-21swapper/121:37:161
389299204,15cyclictest28737-21fschecks_count23:05:150
389699190,3cyclictest0-21swapper/122:12:171
389299197,6cyclictest27935-21munin-run20:00:000
3892991916,2cyclictest94-21kswapd021:38:580
3903991814,2cyclictest39-21ksoftirqd/321:53:143
390399180,2cyclictest141rcu_preempt23:20:073
390199180,18cyclictest0-21swapper/221:57:332
390199180,17cyclictest0-21swapper/222:32:462
389699180,0cyclictest0-21swapper/122:23:041
390199175,1cyclictest141rcu_preempt21:34:022
390199170,1cyclictest0-21swapper/223:25:262
390199170,17cyclictest0-21swapper/223:54:302
390199170,16cyclictest0-21swapper/223:47:162
390199162,13cyclictest0-21swapper/222:20:222
390199160,16cyclictest0-21swapper/200:11:022
390199160,16cyclictest0-21swapper/200:07:112
390199160,11cyclictest0-21swapper/222:05:142
3896991612,0cyclictest0-21swapper/121:26:271
389699161,15cyclictest0-21swapper/122:56:431
389699160,15cyclictest0-21swapper/122:42:211
247482160,0sleep00-21swapper/021:25:290
390399159,2cyclictest141rcu_preempt22:03:253
390399158,2cyclictest225-21systemd-journal00:00:013
390399158,2cyclictest225-21systemd-journal00:00:003
3901991515,0cyclictest0-21swapper/220:25:132
3896991514,1cyclictest0-21swapper/122:47:501
389699150,2cyclictest0-21swapper/123:05:281
389699150,15cyclictest0-21swapper/121:15:501
389299153,6cyclictest22486-21ssh00:00:000
389299153,6cyclictest22486-21ssh00:00:000
390399142,9cyclictest29262-21grep23:25:133
3903991412,2cyclictest39-21ksoftirqd/322:16:223
3903991411,2cyclictest30544-21cat21:29:593
3903991410,3cyclictest23501-21ssh23:01:493
390199148,2cyclictest31806-21grep21:10:272
3901991410,3cyclictest24196-21head19:50:182
3896991410,3cyclictest809-21grep00:05:301
3896991410,2cyclictest22195-21tr19:45:191
389699140,4cyclictest0-21swapper/100:13:021
389699140,14cyclictest0-21swapper/123:58:371
389699140,14cyclictest0-21swapper/123:58:361
389699140,12cyclictest0-21swapper/121:41:201
389299142,7cyclictest0-21swapper/000:15:100
3892991411,2cyclictest11836-21cron23:15:000
3892991410,2cyclictest18376-21wc20:45:190
64282130,1sleep20-21swapper/222:13:202
390399139,3cyclictest9827-21cat20:30:003
390399139,2cyclictest8729-21rm00:10:213
390399139,2cyclictest28362-21tr23:05:143
390399139,2cyclictest10593-21grep19:20:243
390399138,3cyclictest25029-21ls00:40:003
390399133,6cyclictest23022-21apt-get00:00:143
390399133,6cyclictest13703-21ssh21:19:593
3903991310,2cyclictest22175-21readlink20:55:003
3903991310,2cyclictest17645-21cpu22:20:153
3903991310,2cyclictest15786-21cat19:34:593
390199139,4cyclictest18772-21apache200:04:112
390199139,3cyclictest3658-21tr20:15:172
3901991310,2cyclictest29776-21sh23:45:002
3901991310,2cyclictest21076-21ls23:00:202
3901991310,2cyclictest20126-21tr20:50:142
3901991310,2cyclictest17662-21grep20:45:152
3901991310,2cyclictest15381-21wc21:20:202
390199130,13cyclictest0-21swapper/222:01:202
390199130,12cyclictest0-21swapper/222:54:552
389699139,3cyclictest9665-21chrt22:54:051
3896991310,2cyclictest9639-21head00:30:171
3896991310,2cyclictest27245-21grep21:05:131
3896991310,2cyclictest24213-21grep19:50:171
3896991310,2cyclictest21576-21grep23:20:181
389699130,1cyclictest29328-21diskmemload00:01:591
389699130,1cyclictest0-21swapper/123:26:331
389699130,13cyclictest0-21swapper/123:32:391
389699130,13cyclictest0-21swapper/100:38:461
389699130,10cyclictest0-21swapper/100:20:151
389299139,3cyclictest12743-21wc22:55:210
389299135,3cyclictest755-21sh22:29:350
3892991310,2cyclictest26009-21ssh00:20:250
3892991310,2cyclictest25432-21cut21:00:160
3892991310,2cyclictest23575-21tr20:55:190
389299130,2cyclictest0-21swapper/023:19:460
390399129,2cyclictest5004-21tail22:50:293
390399129,2cyclictest3689-21grep20:15:183
390399129,2cyclictest16852-21grep00:15:153
390399129,2cyclictest11918-21wc23:15:003
390399129,2cyclictest10439-21grep22:35:143
390399128,3cyclictest7016-21grep20:20:253
390399128,3cyclictest6413-21cron21:34:593
390399128,3cyclictest16599-21ssh21:40:203
390399128,3cyclictest12315-21chrt19:25:203
390399124,4cyclictest225-21systemd-journal22:50:003
390399124,4cyclictest225-21systemd-journal21:40:003
3903991210,1cyclictest0-21swapper/322:25:233
390199129,2cyclictest9092-21cut00:30:152
390199129,2cyclictest27927-21grep21:05:162
390199129,2cyclictest27074-21cat22:25:192
390199129,2cyclictest22193-21ssh23:20:222
390199129,2cyclictest18011-21munin-run22:40:002
390199129,2cyclictest17714-21grep19:35:272
390199129,2cyclictest14624-21grep23:55:122
390199129,2cyclictest14624-21grep23:55:112
390199128,3cyclictest15610-21sed20:40:172
390199124,4cyclictest0-21swapper/223:05:352
3901991211,1cyclictest0-21swapper/222:55:132
3901991210,1cyclictest24605-21head21:45:192
390199120,1cyclictest0-21swapper/221:53:202
390199120,12cyclictest0-21swapper/223:34:242
389699129,2cyclictest7607-21cat21:55:011
389699129,2cyclictest29328-21diskmemload21:46:441
389699129,2cyclictest25551-21cut22:05:181
389699129,2cyclictest2279-21grep00:25:271
389699129,2cyclictest16672-21cut19:35:171
389699129,2cyclictest15563-21grep20:40:161
389699128,3cyclictest22564-21cut23:40:151
389699128,3cyclictest18825-21grep19:40:141
389699121,6cyclictest9259-21grep22:15:161
3896991212,0cyclictest0-21swapper/122:03:011
389699120,11cyclictest10949-21taskset19:24:201
389699120,10cyclictest0-21swapper/123:00:281
389299129,2cyclictest9665-21grep19:20:150
389299129,2cyclictest4532-21sed20:15:260
389299129,2cyclictest3342-21tr22:50:170
389299129,2cyclictest25695-21grep22:05:190
389299129,2cyclictest22772-21grep19:45:250
389299129,2cyclictest21603-21grep23:00:240
389299129,2cyclictest14084-21tail20:35:260
389299128,3cyclictest9302-21sed20:25:250
389299128,3cyclictest30807-21tr20:05:160
389299128,3cyclictest2473-21cron22:50:010
389299128,2cyclictest13795-21tr23:35:150
389299127,5cyclictest18772-21apache223:46:470
389299124,6cyclictest2931-21runrttasks21:51:270
389299124,5cyclictest5789-21apt-get21:15:150
389299124,5cyclictest481-21gdbus00:09:570
389299124,4cyclictest225-21systemd-journal00:05:010
389299120,11cyclictest24541-21tune2fs21:45:190
390399118,2cyclictest9621-21grep00:30:173
390399118,2cyclictest9267-21ntpq21:55:193
390399118,2cyclictest7189-21ssh23:30:293
390399118,2cyclictest4789-21chrt22:31:113
390399118,2cyclictest31473-21basename00:05:163
390399118,2cyclictest30975-21sort23:45:183
390399118,2cyclictest29553-21ssh21:10:023
390399118,2cyclictest27330-21ssh00:21:093
390399118,2cyclictest25500-21cut21:00:153
390399118,2cyclictest25341-21taskset19:52:293
390399118,2cyclictest24572-21ssh22:05:013
390399118,2cyclictest24066-21sh22:43:023
390399118,2cyclictest23787-21sh21:45:163
390399118,2cyclictest21812-21cat23:40:103
390399118,2cyclictest18137-21sh19:40:003
390399118,2cyclictest17993-21df20:45:153
390399118,2cyclictest16954-21ssh21:20:573
390399118,2cyclictest14916-21rm23:15:523
390399118,2cyclictest1464-21wc00:25:193
390399118,2cyclictest13346-21taskset22:55:283
390399117,3cyclictest22505-21grep20:55:133
390399117,3cyclictest1161-21taskset20:10:173
390399112,3cyclictest3225-21rm22:10:493
390399110,10cyclictest27805-21chrt19:58:403
390199119,1cyclictest24870-21tr21:00:132
390199119,1cyclictest16254-21grep21:40:182
390199118,2cyclictest9393-21ssh21:35:482
390199118,2cyclictest7334-21ssh00:29:182
390199118,2cyclictest6865-21ssh23:11:072
390199118,2cyclictest6454-21tr20:20:202
390199118,2cyclictest29635-21rm21:29:202
390199118,2cyclictest20791-21cron19:45:002
390199118,2cyclictest19211-21expr22:40:152
390199118,2cyclictest18767-21systemctl00:35:252
390199118,2cyclictest14455-21sh23:35:192
390199118,2cyclictest14246-21grep19:30:142
390199118,2cyclictest13450-21ssh23:15:202
390199112,5cyclictest225-21systemd-journal19:49:592
390199111,9cyclictest0-21swapper/200:21:312
389699118,2cyclictest6787-21tr19:15:151
389699118,2cyclictest5851-21sed19:10:291
389699118,2cyclictest527-21grep21:30:261
389699118,2cyclictest32612-21chrt20:09:091
389699118,2cyclictest26599-21tail19:55:171
389699118,2cyclictest21983-21chrt20:55:001
389699118,2cyclictest18120-21grep20:45:171
389699118,2cyclictest16666-21sh23:36:221
389699118,2cyclictest14516-21ssh21:20:141
389699118,2cyclictest13491-21wc23:15:211
389699118,2cyclictest13163-21grep20:35:151
389699118,2cyclictest11162-21ssh21:56:201
389699117,2cyclictest29328-21diskmemload23:51:241
389699112,4cyclictest0-21swapper/120:55:121
389299118,2cyclictest9869-21grep22:15:180
389299118,2cyclictest8868-21grep00:30:140
389299118,2cyclictest6179-21ssh23:50:130
389299118,2cyclictest5959-21cut20:20:160
389299118,2cyclictest5355-21ssh00:27:460
389299118,2cyclictest5024-21ssh23:30:080
389299118,2cyclictest4839-21grep19:10:200
389299118,2cyclictest4272-21ssh21:33:180
389299118,2cyclictest30765-21idleruntime21:10:180
389299118,2cyclictest29904-21grep20:00:250
389299118,2cyclictest28334-21ntp_kernel_pll_21:05:200
389299118,2cyclictest28187-21ssh23:24:380
389299118,2cyclictest24203-21ssh23:40:410
389299118,2cyclictest24200-21sed19:50:170
389299118,2cyclictest20006-21ssh22:20:500
389299118,2cyclictest19331-21cut19:40:160
389299118,2cyclictest18917-21rm21:22:260
389299118,2cyclictest17801-21ssh22:39:500
389299118,2cyclictest17292-21tail22:00:170
389299118,2cyclictest17027-21date20:45:000
389299118,2cyclictest15744-21chrt19:34:580
389299118,2cyclictest1562-21ssh22:30:030
389299118,2cyclictest15065-21ssh21:40:020
389299118,2cyclictest1300-21sed20:10:190
389299118,2cyclictest11458-21ssh21:56:330
389299118,2cyclictest10710-21grep20:30:180
389299112,8cyclictest20175-21pmu-power22:40:210
389299110,10cyclictest4287-21ssh22:11:390
389299110,10cyclictest17189-21apache_volume00:35:120
390399108,1cyclictest40882sleep319:10:053
390399108,1cyclictest28008-21hwlatdetect21:05:173
390399108,1cyclictest21049-21basename19:45:113
390399108,1cyclictest18467-21readlink19:40:013
390399108,1cyclictest15223-21expr20:40:153
390399107,2cyclictest31485-21cat20:05:193
390399107,2cyclictest28971-21head20:00:163
390399107,2cyclictest12690-21expr20:35:133
390399103,3cyclictest225-21systemd-journal20:30:563
390199108,1cyclictest721-21wc20:10:152
390199108,1cyclictest32417-21chrt20:06:382
390199108,1cyclictest30234-21taskset20:04:042
390199108,1cyclictest27716-21taskset19:57:302
390199108,1cyclictest24402-21taskset20:58:182
390199108,1cyclictest12016-21taskset20:33:102
390199108,1cyclictest10512-21grep19:20:242
390199107,2cyclictest4237-21tr19:10:182
390199101,2cyclictest225-21systemd-journal19:15:572
390199100,9cyclictest12432-21ntpq19:25:212
389699108,2cyclictest3706-21chrt20:15:171
389699108,1cyclictest26766-21ssh22:25:171
389699108,1cyclictest2291-21sshd20:10:341
389699107,2cyclictest7723-21wget20:25:131
389699100,9cyclictest25956-21ntpq21:00:201
389299108,1cyclictest17846-21taskset19:36:290
389299101,7cyclictest13-21ksoftirqd/019:25:120
39039996,2cyclictest7307-21hostname19:15:173
39019997,1cyclictest12883-21expr20:35:132
38969997,1cyclictest13980-21expr19:30:141
38969997,1cyclictest10948-21cat20:30:191
38929997,1cyclictest8393-21chrt19:17:150
38929997,1cyclictest20849-21sed20:50:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional