You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-27 - 12:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Sat Jun 27, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
939421290,0sleep10-21swapper/100:13:201
3228129975,19sleep20-21swapper/219:07:192
3216229279,9sleep00-21swapper/019:05:450
3233429064,9sleep30-21swapper/319:08:013
3214828464,15sleep10-21swapper/119:05:341
322002590,0sleep00-21swapper/022:11:010
3259699371,34cyclictest389-21kworker/3:000:31:333
3259699370,37cyclictest389-21kworker/3:000:08:443
32589992624,1cyclictest0-21swapper/122:57:481
3258999251,2cyclictest25374-21diskmemload22:21:271
3259699243,2cyclictest16646-21runrttasks22:21:073
3259699240,5cyclictest29990-21kworker/3:122:07:363
3259499247,13cyclictest4956-21ssh23:31:482
3259499240,23cyclictest29352-21strings21:50:182
32596992321,2cyclictest29990-21kworker/3:121:35:243
3259699220,21cyclictest29990-21kworker/3:121:40:033
32589992222,0cyclictest0-21swapper/123:18:251
3259699210,1cyclictest0-21swapper/321:19:563
32589992116,3cyclictest0-21swapper/121:16:051
3258999210,4cyclictest0-21swapper/122:19:031
3258799212,14cyclictest537-21snmpd23:39:020
3259499200,2cyclictest0-21swapper/222:34:272
3258999200,1cyclictest0-21swapper/121:46:551
3258799201,16cyclictest0-21swapper/022:26:160
3258799200,19cyclictest0-21swapper/000:15:130
3259699190,13cyclictest0-21swapper/320:25:153
32594991918,1cyclictest0-21swapper/200:15:142
3258999195,12cyclictest13485-21/usr/sbin/munin00:15:201
32589991918,1cyclictest0-21swapper/122:50:251
32589991917,2cyclictest25-21ksoftirqd/100:37:341
3258999190,1cyclictest0-21swapper/121:29:361
3258999190,18cyclictest0-21swapper/123:42:301
32587991918,1cyclictest13-21ksoftirqd/000:35:390
32587991918,1cyclictest0-21swapper/023:25:460
3258799190,2cyclictest0-21swapper/022:41:440
3259499182,12cyclictest4870-21fschecks_time21:55:172
3258999181,2cyclictest0-21swapper/122:45:161
3258999180,16cyclictest0-21swapper/123:22:551
3258799180,0cyclictest0-21swapper/021:57:090
3259699171,3cyclictest29990-21kworker/3:122:16:253
3259499174,1cyclictest141rcu_preempt00:34:092
3259499172,2cyclictest22902-21kworker/2:121:17:442
32594991715,1cyclictest0-21swapper/200:01:412
32587991717,0cyclictest0-21swapper/023:34:200
32587991716,1cyclictest0-21swapper/023:47:460
32587991716,1cyclictest0-21swapper/023:47:450
32587991714,2cyclictest13-21ksoftirqd/023:13:170
32596991613,3cyclictest39-21ksoftirqd/322:55:243
3259699160,1cyclictest0-21swapper/322:37:423
3259699160,16cyclictest29990-21kworker/3:121:58:233
3259699160,15cyclictest0-21swapper/300:11:183
3259699160,12cyclictest0-21swapper/322:29:083
3259499162,11cyclictest31589-21apt-get22:50:122
32594991614,2cyclictest0-21swapper/221:25:142
3259499161,13cyclictest11564-21munin-run00:15:002
3259499160,16cyclictest0-21swapper/222:09:312
32589991615,1cyclictest0-21swapper/100:22:371
32589991612,3cyclictest13382-21ls22:00:191
32589991612,3cyclictest13382-21ls22:00:181
3258999160,14cyclictest0-21swapper/123:26:571
3258799164,10cyclictest23769-21cron23:05:000
3258799161,1cyclictest141rcu_preempt21:37:020
3259699150,2cyclictest29990-21kworker/3:122:01:313
3259699150,2cyclictest29990-21kworker/3:122:01:313
3259699150,1cyclictest29990-21kworker/3:122:10:493
3259499155,9cyclictest8542-21sh23:53:432
3259499150,14cyclictest0-21swapper/222:39:542
32589991513,2cyclictest25-21ksoftirqd/121:39:121
32589991513,1cyclictest0-21swapper/121:24:201
3258999150,15cyclictest0-21swapper/122:25:131
3258999150,15cyclictest0-21swapper/122:12:521
3258799155,9cyclictest25868-21awk23:45:000
3258799151,9cyclictest20551-21munin-run20:59:590
3258799151,8cyclictest10813-21apt-get20:40:000
32587991511,3cyclictest8388-21chrt23:15:040
3258799150,1cyclictest0-21swapper/000:27:590
3259699142,5cyclictest0-21swapper/323:58:493
3259699142,1cyclictest0-21swapper/323:13:183
32596991413,1cyclictest0-21swapper/300:35:193
32596991411,2cyclictest7427-21head20:30:203
32596991411,2cyclictest26517-21sed23:25:203
3259499148,5cyclictest537-21snmpd22:22:482
3259499146,2cyclictest0-21swapper/222:10:102
3259499140,14cyclictest0-21swapper/200:36:252
3259499140,12cyclictest0-21swapper/200:26:532
3258999149,4cyclictest0-21swapper/123:46:351
3258999149,4cyclictest0-21swapper/123:46:351
3258999148,2cyclictest31609-21cat22:30:181
3258999148,1cyclictest0-21swapper/121:40:361
3258999145,6cyclictest21780-21apache219:45:201
32589991412,1cyclictest0-21swapper/122:35:271
32589991411,3cyclictest25-21ksoftirqd/121:55:141
32589991410,3cyclictest19684-21wc20:55:211
3258999140,2cyclictest0-21swapper/100:26:001
3258799149,3cyclictest32280-21ls22:30:250
3258799146,7cyclictest15808-21perf20:50:000
3258799142,7cyclictest25374-21diskmemload21:40:130
32587991411,2cyclictest3270-21wc21:15:230
32587991410,3cyclictest18744-21chrt21:25:140
3258799140,8cyclictest14645-21basename19:40:010
3258799140,3cyclictest0-21swapper/000:11:250
3258799140,1cyclictest0-21swapper/020:30:140
3259699139,3cyclictest2245-21sed19:10:253
3259699139,2cyclictest12634-21cut19:35:133
3259699139,2cyclictest11076-21cat20:40:013
32596991310,2cyclictest4110-21tr19:15:203
32596991310,2cyclictest28525-21sleep00:24:593
32596991310,2cyclictest17724-21grep23:20:163
32596991310,2cyclictest10690-21grep21:20:133
3259499139,3cyclictest27242-21grep21:30:132
3259499139,3cyclictest25171-21grep20:00:162
3259499139,3cyclictest21571-21sed21:00:172
3259499139,3cyclictest11679-21cut21:40:142
3259499135,2cyclictest17862-21apache_volume23:40:092
32594991312,1cyclictest10462-21kworker/u8:022:42:502
32594991310,2cyclictest30021-21turbostat22:30:012
32594991310,2cyclictest14926-21grep19:40:142
32594991310,2cyclictest1302-21cron21:14:592
32594991310,2cyclictest12527-21grep22:00:142
32594991310,2cyclictest12527-21grep22:00:132
32594991310,2cyclictest11663-21cat19:30:252
3258999139,3cyclictest25724-21wc20:00:211
3258999139,3cyclictest11109-21wc19:30:191
3258999133,2cyclictest0-21swapper/123:10:131
32589991310,2cyclictest27232-21ssh21:30:131
32589991310,2cyclictest2613-21tr23:50:141
32589991310,2cyclictest19342-21tr00:00:131
3258999130,0cyclictest0-21swapper/100:34:501
3258799139,3cyclictest25455-21wc23:05:190
32587991310,2cyclictest18810-21sed23:20:250
32587991310,2cyclictest18229-21wc20:55:000
3258799130,12cyclictest4711-21sed20:25:170
3259699129,2cyclictest3221-21sed20:20:253
3259699129,2cyclictest28955-21munin-run20:10:013
3259699129,2cyclictest25013-21grep23:05:163
3259699129,2cyclictest17299-21grep19:45:133
3259699129,2cyclictest13052-21grep00:15:173
3259699129,2cyclictest11428-21ssh23:35:263
3259699128,3cyclictest885-21head20:15:263
3259699128,3cyclictest4047-21cat23:50:253
3259699121,1cyclictest0-21swapper/300:27:043
3259699121,0cyclictest0-21swapper/300:01:363
3259499129,2cyclictest9407-21grep20:35:162
3259499129,2cyclictest2308-21head19:10:252
3259499129,2cyclictest20407-21head19:50:162
3259499129,2cyclictest18574-21basename00:00:012
3259499128,3cyclictest23480-21ls22:45:142
3259499128,3cyclictest18461-21sed19:45:222
3259499128,3cyclictest11405-21grep23:35:252
3259499124,4cyclictest225-21systemd-journal20:30:012
3259499121,3cyclictest0-21swapper/219:15:132
32594991211,1cyclictest0-21swapper/221:08:182
32594991210,2cyclictest13874-21ssh21:21:352
32594991210,1cyclictest300592sleep223:46:442
32594991210,1cyclictest300592sleep223:46:442
3259499120,11cyclictest16762-21grep23:00:182
3258999129,2cyclictest9389-21tail19:25:271
3258999129,2cyclictest4617-21cut20:25:171
3258999129,2cyclictest28785-21uniq21:50:151
3258999129,2cyclictest28104-21sed00:05:161
3258999129,2cyclictest21794-21sh19:55:001
3258999128,3cyclictest13681-21apt-get20:45:121
3258999120,11cyclictest0-21swapper/123:30:141
3258799129,2cyclictest7288-21cut22:35:150
3258799129,2cyclictest5639-21grep19:20:140
3258799129,2cyclictest27787-21grep00:05:150
3258799129,2cyclictest19381-21awk19:50:000
3258799129,2cyclictest15138-21tr19:40:150
3258799129,2cyclictest13286-21grep22:20:000
3258799129,2cyclictest12922-21grep20:40:270
3258799128,3cyclictest26669-21grep21:10:160
3258799121,6cyclictest21623-21munin-run22:25:000
32587991212,0cyclictest0-21swapper/000:30:130
3258799120,6cyclictest15400-21sendmail22:59:590
3259699119,1cyclictest24022sleep321:54:053
3259699119,1cyclictest1106-21basename23:30:103
3259699118,2cyclictest39-21ksoftirqd/321:30:343
3259699118,2cyclictest31490-21ssh21:13:153
3259699118,2cyclictest30484-21ssh22:30:113
3259699118,2cyclictest24185-21run-parts20:00:013
3259699118,2cyclictest23717-21ssh23:43:203
3259699118,2cyclictest23122-21ssh21:46:293
3259699118,2cyclictest22894-21taskset21:04:003
3259699118,2cyclictest18749-21expr20:55:143
3259699118,2cyclictest17549-21ssh22:40:583
3259699118,2cyclictest17148-21wc19:45:003
3259699118,2cyclictest15738-21ssh23:00:053
3259699118,2cyclictest14191-21ssh23:18:153
3259699117,3cyclictest26689-21cat23:45:143
3259699117,3cyclictest26689-21cat23:45:133
3259699114,6cyclictest0-21swapper/322:45:203
3259699110,5cyclictest0-21swapper/321:25:253
3259699110,10cyclictest3616-21sshd22:52:203
3259499118,2cyclictest8568-21ssh23:15:122
3259499118,2cyclictest8540-21wc20:35:002
3259499118,2cyclictest5636-21grep22:15:122
3259499118,2cyclictest4500-21ssh23:12:142
3259499118,2cyclictest31847-21grep20:15:142
3259499118,2cyclictest30760-21wc20:10:252
3259499118,2cyclictest30760-21wc20:10:252
3259499118,2cyclictest28772-21cut00:05:212
3259499118,2cyclictest2759-21rm21:35:012
3259499118,2cyclictest26482-21sed23:25:202
3259499118,2cyclictest24192-21ssh23:05:092
3259499118,2cyclictest23770-21ssh21:46:582
3259499118,2cyclictest21449-21grep00:20:162
3259499118,2cyclictest16150-21wget20:50:132
3259499118,2cyclictest14102-21grep19:35:252
3259499118,2cyclictest11439-21grep20:40:142
3259499118,2cyclictest11054-21sh22:56:362
3259499110,10cyclictest16734-21chrt23:20:032
3259499110,10cyclictest14113-21cat20:45:142
3258999118,2cyclictest9338-21ssh23:35:031
3258999118,2cyclictest4897-21taskset19:17:391
3258999118,2cyclictest3673-21cron20:25:001
3258999118,2cyclictest27703-21cat21:10:251
3258999118,2cyclictest21630-21head22:05:171
3258999117,2cyclictest16127-21basename20:50:131
3258999117,1cyclictest25374-21diskmemload23:01:111
3258999113,4cyclictest225-21systemd-journal20:35:201
3258999113,2cyclictest225-21systemd-journal23:56:111
3258999111,3cyclictest17854-21sshd22:41:131
3258999110,10cyclictest6032-21cut19:20:161
3258999110,10cyclictest0-21swapper/123:05:131
3258799118,2cyclictest9683-21ssh23:54:370
3258799118,2cyclictest619-21ssh21:52:420
3258799118,2cyclictest3651-21fschecks_time19:15:170
3258799118,2cyclictest28033-21pluginstate20:05:200
3258799118,2cyclictest26286-21ssh22:46:150
3258799118,2cyclictest23462-21cat21:05:140
3258799118,2cyclictest22738-21taskset21:01:570
3258799118,2cyclictest22252-21ssh22:05:210
3258799118,2cyclictest22039-21taskset00:01:090
3258799118,2cyclictest21948-21ssh00:20:190
3258799118,2cyclictest19719-21ssh21:45:120
3258799118,2cyclictest12774-21grep22:00:140
3258799118,2cyclictest12774-21grep22:00:140
3258799118,2cyclictest12534-21sed23:55:250
3258799118,2cyclictest1115-21grep22:50:260
3258799118,2cyclictest11120-21grep21:20:150
3258799118,2cyclictest1014-21expr19:10:150
3258799111,8cyclictest27120-21apt-get21:30:120
3259699108,1cyclictest8516-21head19:25:183
3259699108,1cyclictest252772sleep320:00:173
3259699108,1cyclictest23569-21cat21:05:143
3259699108,1cyclictest18215-21cat20:55:003
3259699107,2cyclictest29607-21cat20:10:143
3259699107,2cyclictest29607-21cat20:10:143
3259699107,2cyclictest19602-21taskset19:50:083
3259699107,2cyclictest10879-21head19:30:183
3259699100,9cyclictest13298-21taskset20:43:393
3259499108,1cyclictest2249-21sort20:20:152
3259499107,2cyclictest5636-21cut19:20:152
3259499107,2cyclictest28234-21seq20:05:232
3259499107,2cyclictest22818-21grep19:55:182
3259499107,2cyclictest18425-21apt20:55:122
3258999108,1cyclictest23782-21sed19:55:271
3258999108,1cyclictest1530-21uname19:10:191
3258999108,1cyclictest15272-21expr19:40:151
3258999107,2cyclictest27591-21cut20:05:161
3258999107,2cyclictest13215-21grep19:35:161
3258999107,2cyclictest1176-21taskset20:18:461
3258999107,2cyclictest0-21swapper/120:10:121
3258999107,2cyclictest0-21swapper/120:10:121
3258999105,4cyclictest8076-21taskset20:30:351
3258999101,5cyclictest225-21systemd-journal21:04:031
3258999101,4cyclictest11975-21fschecks_time20:40:171
3258799108,1cyclictest95322sleep019:25:560
3258799108,1cyclictest1577-21apt-get20:20:110
3258799108,1cyclictest11999-21taskset19:32:100
3258799107,2cyclictest32006-21expr20:15:150
3258799107,2cyclictest31155-21taskset20:12:580
3258799107,2cyclictest31155-21taskset20:12:570
3258799107,2cyclictest23686-21sed19:55:260
3258799107,2cyclictest21550-21chrt19:52:370
3258799106,3cyclictest24705-21tr20:00:140
325969998,1cyclictest15683-21taskset20:48:503
325969997,1cyclictest7641-21cat19:25:013
325949997,1cyclictest96642sleep219:27:392
325899992,3cyclictest225-21systemd-journal21:09:591
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional