You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-17 - 03:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Mon Aug 17, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3133029355,34sleep00-21swapper/019:06:000
3145129164,9sleep20-21swapper/219:07:342
3131628567,13sleep10-21swapper/119:05:491
3161228269,9sleep30-21swapper/319:09:403
31730993026,3cyclictest9637-21perf21:40:003
67962280,0sleep20-21swapper/221:18:412
31730992524,1cyclictest0-21swapper/323:52:193
3172699250,22cyclictest0-21swapper/200:06:552
31730992421,2cyclictest32707-21systemd-cgroups00:08:583
3172199240,22cyclictest0-21swapper/119:50:211
31730992321,1cyclictest24318-21diskmemload21:22:503
31730992313,5cyclictest0-21swapper/323:56:333
3172699230,22cyclictest0-21swapper/200:25:352
3172699230,0cyclictest0-21swapper/200:11:162
3171699238,6cyclictest225-21systemd-journal22:00:000
31730992219,2cyclictest24318-21diskmemload21:17:433
31716992222,0cyclictest0-21swapper/022:01:400
3173099214,12cyclictest0-21swapper/319:15:113
31730992119,1cyclictest24318-21diskmemload23:39:063
31730992119,1cyclictest24318-21diskmemload21:30:583
31730992119,1cyclictest0-21swapper/323:03:083
3172699210,4cyclictest0-21swapper/222:04:222
3172699210,18cyclictest0-21swapper/221:47:402
3172199212,17cyclictest0-21swapper/121:48:091
3172199210,1cyclictest0-21swapper/119:25:151
3172199210,1cyclictest0-21swapper/119:25:151
3171699210,21cyclictest0-21swapper/021:37:150
31730992018,1cyclictest24318-21diskmemload23:31:193
31730992016,3cyclictest24318-21diskmemload23:10:163
31730992016,3cyclictest24318-21diskmemload23:10:153
31730992014,5cyclictest0-21swapper/319:20:153
3173099201,18cyclictest0-21swapper/321:46:193
3172699200,17cyclictest0-21swapper/222:39:012
31721992018,1cyclictest0-21swapper/121:40:181
3172199200,3cyclictest0-21swapper/123:10:151
3172199200,3cyclictest0-21swapper/123:10:151
3172199200,20cyclictest0-21swapper/100:12:141
31730991917,1cyclictest3330-21apache_volume00:30:163
31730991917,1cyclictest12250-21sh00:35:173
31730991915,4cyclictest24318-21diskmemload00:20:433
31730991915,1cyclictest24318-21diskmemload23:45:373
31730991914,1cyclictest24318-21diskmemload00:14:463
31730991911,6cyclictest24318-21diskmemload22:45:473
31726991918,1cyclictest0-21swapper/219:10:182
3172699190,19cyclictest0-21swapper/223:32:482
31721991916,2cyclictest25-21ksoftirqd/123:05:251
31716991917,1cyclictest0-21swapper/022:45:300
31730991816,1cyclictest25684-21latency_hist00:05:003
31730991816,1cyclictest24318-21diskmemload23:05:393
31730991816,1cyclictest0-21swapper/320:30:143
31730991815,2cyclictest24318-21diskmemload21:11:523
31730991814,4cyclictest0-21swapper/320:40:283
31730991814,3cyclictest0-21swapper/322:05:163
31726991817,1cyclictest0-21swapper/221:58:392
3172699180,18cyclictest0-21swapper/223:41:052
3172699180,18cyclictest0-21swapper/200:30:542
31721991817,1cyclictest0-21swapper/122:38:231
3172199180,18cyclictest0-21swapper/122:42:341
3172199180,0cyclictest0-21swapper/121:21:591
3173099179,4cyclictest8161-21grep20:35:143
31730991716,1cyclictest0-21swapper/319:30:133
31730991715,1cyclictest5445-21perf22:55:013
31730991715,1cyclictest30269-21grep20:14:593
31730991714,1cyclictest24318-21diskmemload22:30:393
31730991713,3cyclictest28820-21if_enp2s022:10:183
31730991713,3cyclictest0-21swapper/321:50:213
31730991712,4cyclictest0-21swapper/321:55:153
31721991716,1cyclictest0-21swapper/123:17:431
3172199170,13cyclictest24318-21diskmemload21:57:031
3171699170,2cyclictest0-21swapper/021:21:130
3171699170,15cyclictest0-21swapper/022:56:100
31730991615,1cyclictest24318-21diskmemload21:29:553
31730991615,1cyclictest0-21swapper/323:40:133
31730991614,2cyclictest0-21swapper/323:20:123
31730991612,2cyclictest0-21swapper/300:25:563
31726991613,2cyclictest8482-21munin-node23:39:442
3172699161,14cyclictest0-21swapper/223:51:392
3172699160,15cyclictest0-21swapper/222:05:572
3172699160,15cyclictest0-21swapper/221:32:522
3172199163,6cyclictest24318-21diskmemload00:20:241
31721991615,1cyclictest0-21swapper/123:01:031
31721991615,1cyclictest0-21swapper/122:08:141
31721991615,1cyclictest0-21swapper/100:05:431
3171699166,2cyclictest225-21systemd-journal23:49:590
3171699161,7cyclictest23138-21munin-run20:00:000
3171699160,0cyclictest0-21swapper/023:22:080
31730991513,1cyclictest24318-21diskmemload23:15:393
31730991513,1cyclictest24318-21diskmemload22:40:193
31730991513,1cyclictest24318-21diskmemload22:25:033
31730991513,1cyclictest0-21swapper/322:24:023
31730991512,3cyclictest0-21swapper/319:40:143
31730991511,4cyclictest0-21swapper/322:58:543
31730991510,5cyclictest0-21swapper/321:44:273
3172699151,13cyclictest17275-21cron23:59:592
3172199159,4cyclictest16757-21grep19:45:171
3172199158,6cyclictest12558-21cat20:45:001
3172199154,7cyclictest2807-21/usr/sbin/munin21:35:191
31721991514,1cyclictest0-21swapper/121:53:031
31721991513,2cyclictest25-21ksoftirqd/122:53:031
31721991512,2cyclictest0-21swapper/122:11:491
31721991511,3cyclictest4572-21sort22:15:171
31721991510,3cyclictest19531-21sh21:00:011
3171699151,8cyclictest12422-21munin-run20:45:000
31716991513,2cyclictest13-21ksoftirqd/020:15:130
31716991511,3cyclictest19619-21grep22:05:160
3171699150,14cyclictest0-21swapper/000:22:360
3173099149,4cyclictest19261-21fschecks_count19:50:163
3173099147,4cyclictest24318-21diskmemload22:03:153
31730991413,1cyclictest0-21swapper/300:15:173
3172699149,4cyclictest25650-21grep21:10:182
3172699145,5cyclictest225-21systemd-journal20:05:002
31726991411,2cyclictest16186-21wc19:45:002
3172699140,1cyclictest0-21swapper/221:42:392
3172699140,14cyclictest0-21swapper/223:08:222
3172699140,12cyclictest0-21swapper/221:23:082
3172199141,8cyclictest38150irq/131-snd_hda00:15:131
31721991410,3cyclictest20652-21ssh22:25:141
3172199140,13cyclictest0-21swapper/121:20:001
3171699141,6cyclictest24330-21awk21:10:000
31716991410,3cyclictest17776-21wc19:45:270
3171699140,1cyclictest0-21swapper/021:26:360
3171699140,14cyclictest0-21swapper/000:17:250
3173099135,1cyclictest0-21swapper/320:50:133
31730991310,2cyclictest5541-21threads22:15:263
31730991310,2cyclictest29400-21sh23:28:083
31730991310,2cyclictest21011-21apache_processe19:55:133
3172699139,3cyclictest22007-21ssh00:21:192
31726991310,2cyclictest7577-21cron20:35:012
31726991310,2cyclictest31969-21cut23:10:192
31726991310,2cyclictest31969-21cut23:10:182
31726991310,2cyclictest30706-21grep22:50:162
31726991310,2cyclictest11667-21cron22:20:012
3172699130,13cyclictest0-21swapper/223:22:242
3172699130,13cyclictest0-21swapper/222:27:182
3172699130,12cyclictest29642-21cat22:10:252
3172699130,12cyclictest25376-21cron21:30:002
3172199139,3cyclictest27528-21ssh23:26:411
31721991310,2cyclictest6413-21cron19:25:001
31721991310,2cyclictest22145-21awk19:55:221
3172199130,12cyclictest24452-21cron23:45:001
3171699139,3cyclictest32734-21wc21:15:000
3171699139,3cyclictest2694-21wc20:25:010
3171699139,3cyclictest24538-21grep23:45:000
3171699139,2cyclictest30927-21wc23:10:000
3171699139,2cyclictest22329-21wc22:25:280
3171699132,5cyclictest0-21swapper/022:32:520
31716991311,2cyclictest13-21ksoftirqd/021:31:420
3171699130,1cyclictest0-21swapper/022:39:000
3171699130,1cyclictest0-21swapper/022:10:130
3171699130,11cyclictest17410-21wc00:00:000
3173099129,2cyclictest7307-21cut19:25:193
3173099129,2cyclictest7307-21cut19:25:183
3173099129,2cyclictest4999-21munin-plugin-st20:30:003
3173099129,2cyclictest30813-21tr20:15:153
3173099128,3cyclictest19130-21sed20:55:283
3173099128,3cyclictest0-21swapper/322:35:093
31730991210,1cyclictest2204-21grep20:20:273
31730991210,1cyclictest0-21swapper/320:00:233
3172699129,2cyclictest7928-21tr19:25:232
3172699129,2cyclictest7928-21tr19:25:232
3172699129,2cyclictest3564-21grep19:15:262
3172699129,2cyclictest24743-21grep23:25:202
3172699129,2cyclictest2348-21sed21:35:152
3172699128,3cyclictest4567-21grep19:20:162
3172699128,3cyclictest11920-21cut00:35:162
3172699127,4cyclictest13988-21cat22:20:262
3172699120,12cyclictest0-21swapper/200:03:192
3172699120,10cyclictest0-21swapper/223:03:552
3172199129,2cyclictest6253-21grep20:30:191
3172199129,2cyclictest31655-21tr20:15:201
3172199129,2cyclictest30413-21wc20:15:001
3172199129,2cyclictest27883-21sed21:30:291
3172199129,2cyclictest2152-21tr19:15:141
3172199129,2cyclictest16284-21sort23:20:151
3172199129,2cyclictest13600-21sed00:35:281
3172199129,2cyclictest12413-21uname22:00:211
3172199128,3cyclictest20313-21grep21:00:171
3172199128,3cyclictest17460-21sh21:25:071
31721991212,0cyclictest0-21swapper/123:57:561
31721991211,1cyclictest0-21swapper/100:01:531
31721991210,2cyclictest25-21ksoftirqd/122:33:061
3172199120,12cyclictest0-21swapper/122:57:291
3171699129,2cyclictest784-21wc19:10:220
3171699129,2cyclictest32123-21head23:10:180
3171699129,2cyclictest32123-21head23:10:180
3171699129,2cyclictest15698-21grep20:50:160
3171699129,2cyclictest14990-21expr23:00:170
3171699129,2cyclictest12935-21cut22:20:170
3171699129,2cyclictest12423-21head00:35:190
3171699129,2cyclictest10062-21ls00:15:000
3171699128,3cyclictest32655-21sshd21:53:280
3171699128,3cyclictest21143-21cron22:45:010
3171699128,2cyclictest6412-21cron19:25:000
3171699128,2cyclictest11420-21ssh21:40:200
3171699128,1cyclictest0-21swapper/023:37:480
3171699121,5cyclictest0-21swapper/023:32:480
3171699120,1cyclictest0-21swapper/000:08:370
3171699120,11cyclictest1231-21ssh21:15:150
3171699120,10cyclictest0-21swapper/023:53:530
3173099118,2cyclictest27822-21taskset20:09:163
3173099117,3cyclictest12309-21ls19:35:163
3173099117,3cyclictest0-21swapper/319:10:183
3173099112,7cyclictest23846-21unixbench_multi21:05:263
3173099111,5cyclictest225-21systemd-journal19:50:003
3172699118,2cyclictest9767-21chrt20:36:532
3172699118,2cyclictest8214-21ssh23:15:202
3172699118,2cyclictest3312-21expr20:25:152
3172699118,2cyclictest29074-21ssh21:50:402
3172699118,2cyclictest28913-21sed20:10:182
3172699118,2cyclictest28010-21date20:10:012
3172699118,2cyclictest21768-21sed19:55:192
3172699118,2cyclictest17715-21grep20:55:162
3172699118,2cyclictest11867-21ntpq00:15:212
3172699118,2cyclictest10469-21ssh22:57:232
3172699118,1cyclictest1431-21awk20:20:192
3172699111,8cyclictest22495-21gltestperf22:45:182
3172699110,2cyclictest0-21swapper/222:30:452
3172699110,1cyclictest0-21swapper/222:42:332
3172699110,11cyclictest0-21swapper/223:45:032
3172199118,2cyclictest8623-21grep23:35:161
3172199118,2cyclictest8242-21expr20:35:151
3172199118,2cyclictest743-21grep23:30:191
3172199118,2cyclictest640-21grep20:20:151
3172199118,2cyclictest5069-21ssh00:30:281
3172199118,2cyclictest32096-21tr19:10:151
3172199118,2cyclictest28810-21tr00:25:231
3172199118,2cyclictest28260-21ssh21:11:341
3172199118,2cyclictest26410-21sort20:05:171
3172199118,2cyclictest26210-21sort23:45:171
3172199118,2cyclictest23072-21ls22:45:221
3172199118,2cyclictest1435-21grep23:50:161
3172199118,2cyclictest14179-21grep19:40:151
3172199118,2cyclictest13462-21sed20:45:171
3172199118,2cyclictest12951-21cat19:35:241
3172199117,1cyclictest241792sleep121:08:341
3171699118,2cyclictest8599-21ssh22:17:350
3171699118,2cyclictest3161-21ssh00:30:090
3171699118,2cyclictest29828-21ssh00:25:530
3171699118,2cyclictest25550-21ssh21:49:260
3171699118,2cyclictest24837-21cat20:00:270
3171699118,2cyclictest2256-21ssh22:52:300
3171699118,2cyclictest20932-21cat19:55:010
3171699118,2cyclictest20272-21seq00:00:480
3171699118,2cyclictest11548-21ssh23:17:150
3171699118,2cyclictest10773-21sed19:30:250
3171699117,3cyclictest12715-21tr19:35:200
3171699117,2cyclictest14649-21tail19:40:170
3171699111,5cyclictest0-21swapper/023:25:170
3173099108,1cyclictest12997-21expr20:45:153
3173099107,2cyclictest20042-21sort21:00:163
3172699108,1cyclictest21448-21unixbench_multi21:00:282
3172699108,1cyclictest205462sleep219:52:332
3172699108,1cyclictest133442sleep219:36:462
3172699107,2cyclictest22323-21cut21:05:152
3172699107,2cyclictest17028-21grep19:45:192
3172699107,2cyclictest16945-21taskset20:52:222
3172699107,2cyclictest138392sleep220:45:222
3172699107,2cyclictest12425-21basename20:45:012
3172199108,1cyclictest23797-21ls20:00:171
3172199108,1cyclictest14407-21ssh22:20:331
3172199107,2cyclictest9535-21expr19:30:141
3171699108,1cyclictest8645-21expr20:35:170
3171699108,1cyclictest1921-21dpkg19:15:110
3171699108,1cyclictest12910-21expr20:45:160
3171699107,2cyclictest4093-21cut20:25:220
3171699107,2cyclictest30079-21taskset20:12:440
3171699107,2cyclictest27626-21chrt20:06:410
3171699107,2cyclictest19824-21apt21:00:140
3171699107,2cyclictest17640-21ls20:55:170
3171699103,3cyclictest273-21systemd-udevd20:32:040
317269998,1cyclictest9402-21expr19:30:142
317269997,1cyclictest30792-21expr20:15:142
317219990,8cyclictest16746-21grep20:50:261
317169997,1cyclictest6998-21cstates19:25:160
317169997,1cyclictest6998-21cstates19:25:160
317219984,2cyclictest3125-21tr20:25:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional