You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-16 - 07:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot6s.osadl.org (updated Tue Sep 16, 2025 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
31875210683,18sleep00-21swapper/019:08:080
31988210584,16sleep30-21swapper/319:09:353
31802210585,15sleep10-21swapper/119:07:121
31783210578,6sleep20-21swapper/219:06:572
79602730,0sleep20-21swapper/219:30:002
3204899341,2cyclictest0-21swapper/022:40:140
68362240,1sleep00-21swapper/020:45:180
50332240,1sleep229-21rcuc/222:05:042
3205199240,22cyclictest15941-21diskmemload22:55:013
3204899231,3cyclictest0-21swapper/021:27:520
3204899230,21cyclictest0-21swapper/021:05:110
3204899230,21cyclictest0-21swapper/021:05:110
71042220,1sleep1221ktimersoftd/122:40:141
3204899221,19cyclictest5129-21runrttasks21:23:130
3204899221,19cyclictest10115-21ssh22:10:130
3205099210,5cyclictest0-21swapper/222:21:042
3204899211,2cyclictest0-21swapper/022:30:120
32048992110,9cyclictest8489-21fschecks_count19:30:150
3204899210,3cyclictest25329-21mailstats20:10:270
3204899210,2cyclictest0-21swapper/022:36:010
3205199200,18cyclictest0-21swapper/320:25:123
3204899201,17cyclictest0-21swapper/023:10:020
3204899200,18cyclictest0-21swapper/022:15:130
3204899200,11cyclictest0-21swapper/023:38:540
25522200,0sleep20-21swapper/222:01:392
3205099192,2cyclictest0-21swapper/222:26:042
32050991915,3cyclictest9627-21kworker/2:123:20:132
3204999190,4cyclictest24364-21apt-get22:25:041
3204999190,3cyclictest0-21swapper/122:21:041
32048991910,8cyclictest0-21swapper/023:40:070
3204899190,2cyclictest0-21swapper/023:22:390
3204899190,2cyclictest0-21swapper/000:27:340
3204899190,2cyclictest0-21swapper/000:18:170
3204899190,1cyclictest0-21swapper/023:18:270
3204899190,1cyclictest0-21swapper/022:59:010
3204899190,1cyclictest0-21swapper/021:15:550
3204899190,17cyclictest0-21swapper/023:49:340
3204899190,17cyclictest0-21swapper/023:27:390
3204899190,11cyclictest15941-21diskmemload22:22:350
86252180,1sleep18631-21fschecks_time19:30:151
32051991814,2cyclictest0-21swapper/321:50:123
32050991815,2cyclictest5534-21kworker/u8:223:47:052
32050991815,2cyclictest31148-21kworker/u8:023:38:052
32050991815,2cyclictest28301-21kworker/u8:120:34:012
32049991815,2cyclictest7487-21kworker/u8:222:14:031
32049991815,2cyclictest5534-21kworker/u8:200:19:051
32049991815,2cyclictest23317-21kworker/u8:022:57:051
32049991815,2cyclictest22241-21kworker/u8:022:03:041
3204999180,13cyclictest0-21swapper/121:46:031
3204899188,8cyclictest0-21swapper/022:08:580
3204899187,9cyclictest0-21swapper/022:50:250
3204899185,3cyclictest15941-21diskmemload21:50:050
3204899181,15cyclictest32081-21munin-run20:30:000
3204899180,2cyclictest14221-21ssh22:46:370
3204899180,2cyclictest0-21swapper/023:51:330
3204899180,17cyclictest0-21swapper/023:08:490
3204899180,17cyclictest0-21swapper/021:48:570
3204899180,16cyclictest15941-21diskmemload22:25:580
3204899180,10cyclictest0-21swapper/021:55:450
3204899180,10cyclictest0-21swapper/021:36:180
32050991714,2cyclictest5534-21kworker/u8:200:05:042
32050991714,2cyclictest29536-21kworker/u8:222:36:042
32050991714,2cyclictest27778-21kworker/u8:019:32:042
32050991714,2cyclictest20271-21kworker/u8:220:39:042
32050991714,2cyclictest13580-21kworker/u8:019:54:042
32050991714,2cyclictest11548-21kworker/u8:021:27:042
32050991714,2cyclictest10179-21kworker/u8:219:37:032
3205099171,14cyclictest0-21swapper/223:09:052
3205099170,12cyclictest0-21swapper/220:40:042
3204999172,3cyclictest221ktimersoftd/122:38:031
32049991714,2cyclictest27778-21kworker/u8:019:28:011
32049991714,2cyclictest27778-21kworker/u8:019:19:041
32049991714,2cyclictest1881-21kworker/u8:020:48:041
32049991714,2cyclictest15916-21kworker/u8:100:33:051
32049991714,2cyclictest13580-21kworker/u8:019:48:041
32049991714,2cyclictest12461-21kworker/u8:023:28:051
3204999170,2cyclictest15941-21diskmemload22:15:041
3204899177,2cyclictest0-21swapper/021:14:420
3204899173,4cyclictest18855-21grep19:55:160
3204899171,7cyclictest31030-21cron23:05:000
3204899171,6cyclictest0-21swapper/019:15:390
3204899170,7cyclictest4766-21cron22:04:590
3204899170,16cyclictest0-21swapper/021:33:440
3204899170,16cyclictest0-21swapper/000:05:140
32050991613,2cyclictest9182-21kworker/u8:122:53:052
32050991613,2cyclictest5534-21kworker/u8:200:10:042
32050991613,2cyclictest3878-21kworker/u8:219:22:042
32050991613,2cyclictest31148-21kworker/u8:023:57:042
32050991613,2cyclictest31148-21kworker/u8:023:53:052
32050991613,2cyclictest22986-21kworker/u8:223:34:012
32050991613,2cyclictest20271-21kworker/u8:220:18:042
32050991613,2cyclictest20271-21kworker/u8:220:01:042
32050991613,2cyclictest13580-21kworker/u8:020:10:042
32050991613,2cyclictest13580-21kworker/u8:020:09:042
32050991613,2cyclictest13580-21kworker/u8:019:42:042
32050991613,2cyclictest11548-21kworker/u8:021:16:042
3205099161,2cyclictest0-21swapper/221:31:042
3205099161,2cyclictest0-21swapper/221:20:042
32050991612,2cyclictest0-21swapper/222:10:182
3205099160,2cyclictest0-21swapper/222:15:042
3205099160,2cyclictest0-21swapper/221:56:012
32049991613,2cyclictest933-21kworker/u8:119:12:041
32049991613,2cyclictest9182-21kworker/u8:123:04:031
32049991613,2cyclictest5534-21kworker/u8:200:27:041
32049991613,2cyclictest5534-21kworker/u8:200:04:041
32049991613,2cyclictest31148-21kworker/u8:023:55:051
32049991613,2cyclictest22051-21kworker/u8:200:36:011
32049991613,2cyclictest1882-21kworker/u8:122:06:011
32049991613,2cyclictest18454-21kworker/u8:022:46:041
32049991613,2cyclictest12461-21kworker/u8:023:22:041
32049991613,2cyclictest11548-21kworker/u8:021:23:041
3204899164,3cyclictest5129-21runrttasks20:53:230
3204899164,3cyclictest4324-21nscd00:20:130
3204899164,3cyclictest4166-21rsyslogd21:00:000
3204899164,2cyclictest15941-21diskmemload00:35:120
3204899163,4cyclictest20168-21munin-node0
3204899163,3cyclictest607-21grep19:10:170
3204899163,3cyclictest4050-21cpu19:20:140
3204899163,3cyclictest13515-21unixbench_singl21:00:280
3204899161,6cyclictest136962sleep019:42:240
3204899161,6cyclictest0-21swapper/019:45:450
3204899161,4cyclictest31379-21grep00:10:280
3204899161,4cyclictest23760-21fschecks_count23:30:120
3204899161,4cyclictest13574-21ssh21:40:150
3204899161,4cyclictest0-21swapper/000:30:150
3204899161,3cyclictest19817-21cron00:00:000
3204899161,3cyclictest0-21swapper/020:30:120
3204899160,15cyclictest0-21swapper/020:00:160
3205199150,13cyclictest0-21swapper/323:45:253
3205199150,13cyclictest0-21swapper/322:52:163
3205199150,13cyclictest0-21swapper/321:48:483
3205199150,13cyclictest0-21swapper/300:36:013
32050991512,2cyclictest9519-21kworker/u8:120:56:012
32050991512,2cyclictest933-21kworker/u8:119:11:032
32050991512,2cyclictest26098-21kworker/u8:121:45:032
32050991512,2cyclictest20271-21kworker/u8:220:45:042
32050991512,2cyclictest11548-21kworker/u8:021:04:042
32050991510,3cyclictest0-21swapper/200:36:192
3205099150,4cyclictest0-21swapper/222:45:042
3205099150,2cyclictest0-21swapper/223:17:042
3205099150,2cyclictest0-21swapper/222:31:042
3205099150,13cyclictest15114-21ps19:45:192
3204999151,3cyclictest4829-21Xorg21:32:591
32049991512,2cyclictest9519-21kworker/u8:120:51:031
32049991512,2cyclictest31148-21kworker/u8:023:42:041
32049991512,2cyclictest27820-21kworker/u8:000:11:051
32049991512,2cyclictest20271-21kworker/u8:220:38:041
32049991512,1cyclictest0-21swapper/120:25:111
3204999150,2cyclictest0-21swapper/123:17:041
3204999150,2cyclictest0-21swapper/123:10:051
3204999150,2cyclictest0-21swapper/122:31:041
3204999150,2cyclictest0-21swapper/121:56:051
3204999150,13cyclictest29062-21missed_timers23:35:181
3204999150,10cyclictest0-21swapper/120:05:041
3204899154,3cyclictest28314-21aten2.4_r3power20:20:120
3204899153,3cyclictest3889-21aten2.4_r3power20:40:130
3204899153,3cyclictest16174-21aten2.4_r3power19:50:120
3204899153,3cyclictest10381-21grep19:35:130
3204899152,3cyclictest26764-21fschecks_time20:15:150
3204899151,3cyclictest2728-21grep20:35:200
3204899151,3cyclictest0-21swapper/020:05:170
3204899151,3cyclictest0-21swapper/019:25:140
45812140,3sleep30-21swapper/320:40:183
32051991411,2cyclictest0-21swapper/322:48:013
32051991411,2cyclictest0-21swapper/300:05:223
3205199140,3cyclictest0-21swapper/323:01:273
3205199140,2cyclictest0-21swapper/322:10:223
3205199140,13cyclictest0-21swapper/323:40:023
3205199140,12cyclictest6720-21ssh00:19:363
3205199140,12cyclictest15941-21diskmemload00:00:213
3205199140,12cyclictest0-21swapper/322:41:273
3205199140,12cyclictest0-21swapper/322:35:193
32050991411,2cyclictest1881-21kworker/u8:020:50:042
32050991411,2cyclictest13580-21kworker/u8:019:55:042
32050991411,2cyclictest0-21swapper/200:15:192
3205099140,2cyclictest15941-21diskmemload23:42:292
3205099140,2cyclictest0-21swapper/223:26:042
3205099140,2cyclictest0-21swapper/223:10:042
3205099140,13cyclictest0-21swapper/221:12:052
3204999149,3cyclictest0-21swapper/100:22:211
3204999142,6cyclictest0-21swapper/120:55:011
32049991411,2cyclictest7032-21kworker/u8:021:40:061
32049991411,2cyclictest31148-21kworker/u8:023:52:031
32049991411,2cyclictest31148-21kworker/u8:023:45:041
32049991411,2cyclictest22241-21kworker/u8:021:50:041
32049991411,2cyclictest11548-21kworker/u8:021:02:041
3204999140,2cyclictest15941-21diskmemload23:32:091
3204999140,13cyclictest0-21swapper/120:00:031
3204999140,12cyclictest0-21swapper/123:06:001
3204999140,10cyclictest0-21swapper/121:11:041
211732141,3sleep115941-21diskmemload21:15:171
32051991310,2cyclictest0-21swapper/323:55:473
32051991310,2cyclictest0-21swapper/323:51:373
32051991310,2cyclictest0-21swapper/323:17:283
32051991310,2cyclictest0-21swapper/322:27:173
32051991310,2cyclictest0-21swapper/322:15:573
32051991310,2cyclictest0-21swapper/322:07:133
32051991310,2cyclictest0-21swapper/321:05:143
32051991310,2cyclictest0-21swapper/321:05:143
32051991310,2cyclictest0-21swapper/300:30:533
32051991310,2cyclictest0-21swapper/300:27:553
3205199130,2cyclictest8194-21ssh23:14:363
3205199130,2cyclictest15941-21diskmemload00:10:133
3205199130,2cyclictest0-21swapper/323:35:333
3205199130,2cyclictest0-21swapper/322:30:173
3205199130,2cyclictest0-21swapper/321:30:173
3205199130,2cyclictest0-21swapper/321:20:323
3205199130,2cyclictest0-21swapper/321:10:153
3205199130,2cyclictest0-21swapper/319:55:143
3205199130,2cyclictest0-21swapper/300:20:323
3205199130,12cyclictest0-21swapper/321:43:223
3205199130,12cyclictest0-21swapper/321:35:123
3205199130,11cyclictest0-21swapper/321:15:483
3205099139,3cyclictest23482-21ssh00:03:082
32050991310,2cyclictest17349-21ssh21:44:292
32050991310,2cyclictest11213-21ssh22:44:292
32050991310,2cyclictest0-21swapper/200:20:202
3205099130,9cyclictest0-21swapper/222:55:042
3205099130,2cyclictest15941-21diskmemload23:00:162
3205099130,2cyclictest0-21swapper/221:39:042
3205099130,2cyclictest0-21swapper/220:21:042
3205099130,12cyclictest0-21swapper/220:27:042
3205099130,11cyclictest0-21swapper/200:27:292
3204999131,10cyclictest0-21swapper/120:20:141
3204999130,2cyclictest0-21swapper/121:39:041
3204999130,12cyclictest0-21swapper/119:50:191
3204999130,11cyclictest0-21swapper/100:06:461
219392130,3sleep321951-21ssh22:21:163
3205199129,2cyclictest0-21swapper/320:35:273
3205199129,2cyclictest0-21swapper/320:30:183
3205199129,2cyclictest0-21swapper/320:15:133
3205199120,2cyclictest0-21swapper/323:26:373
3205199120,2cyclictest0-21swapper/323:06:033
3205199120,2cyclictest0-21swapper/319:35:243
3205199120,11cyclictest0-21swapper/323:31:383
3205199120,11cyclictest0-21swapper/323:21:333
3205199120,11cyclictest0-21swapper/322:01:133
3205199120,11cyclictest0-21swapper/321:55:123
3205199120,11cyclictest0-21swapper/321:25:163
3205199120,11cyclictest0-21swapper/321:00:023
3205199120,11cyclictest0-21swapper/319:45:173
3205199120,11cyclictest0-21swapper/319:40:153
3205199120,11cyclictest0-21swapper/319:30:143
3205199120,11cyclictest0-21swapper/319:10:273
3205099129,2cyclictest11548-21kworker/u8:021:05:042
3205099129,2cyclictest11548-21kworker/u8:021:05:042
3205099120,2cyclictest0-21swapper/221:52:042
3205099120,2cyclictest0-21swapper/200:30:152
3204999121,9cyclictest32288-21munin-node1
3204999121,3cyclictest19945-21munin-run20:00:001
3204999120,2cyclictest15941-21diskmemload22:51:021
3204999120,2cyclictest0-21swapper/121:28:041
3204999120,2cyclictest0-21swapper/119:35:041
3204999120,2cyclictest0-21swapper/119:24:011
3204999120,10cyclictest0-21swapper/120:15:211
3204999120,10cyclictest0-21swapper/120:10:281
3205199118,2cyclictest0-21swapper/319:25:203
3205199118,2cyclictest0-21swapper/319:20:253
3205199112,7cyclictest0-21swapper/319:18:593
3205199110,2cyclictest0-21swapper/320:51:163
3204999117,2cyclictest0-21swapper/121:05:111
3204999117,2cyclictest0-21swapper/121:05:111
3204999111,7cyclictest0-21swapper/120:44:591
3204999110,2cyclictest0-21swapper/119:40:041
3205199105,3cyclictest5129-21runrttasks20:05:153
3205099100,2cyclictest1596-21apt-get19:15:042
320519994,3cyclictest28637-21df_abs20:20:133
320519992,6cyclictest0-21swapper/320:56:023
320519992,5cyclictest278-21jbd2/sda1-820:47:213
320519990,2cyclictest0-21swapper/320:11:513
320519985,2cyclictest0-21swapper/319:50:183
320519983,3cyclictest0-21swapper/320:00:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional