You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-18 - 09:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot6s.osadl.org (updated Tue Nov 18, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16374211897,15sleep20-21swapper/219:08:542
163342117106,6sleep00-21swapper/019:08:230
163912116106,6sleep30-21swapper/319:09:063
16320210695,6sleep10-21swapper/119:08:121
295702820,1sleep20-21swapper/221:00:182
268612780,1sleep20-21swapper/223:19:352
12022720,1sleep30-21swapper/322:51:543
76152690,1sleep20-21swapper/222:59:152
258752590,1sleep30-21swapper/320:50:263
1648999361,33cyclictest0-21swapper/021:10:150
73732230,1sleep21649199cyclictest21:16:102
224892230,0sleep20-21swapper/200:21:282
224892230,0sleep20-21swapper/200:21:282
219282230,1sleep10-21swapper/120:40:271
50312220,0sleep2301ktimersoftd/222:21:552
16490992118,2cyclictest24352-21kworker/u8:221:43:051
67082200,0sleep20-21swapper/223:31:062
16492992017,2cyclictest32602-21diskmemload00:35:153
1649299200,2cyclictest0-21swapper/319:40:143
1649299200,18cyclictest0-21swapper/321:45:113
1648999200,2cyclictest0-21swapper/020:35:160
222832190,0sleep00-21swapper/023:48:030
1649099193,2cyclictest18151-21ssh23:44:041
16490991916,2cyclictest28905-21kworker/u8:119:47:041
16490991916,2cyclictest26258-21kworker/u8:219:34:011
1649199180,2cyclictest32602-21diskmemload22:39:052
1649099182,2cyclictest23-21ksoftirqd/100:04:051
16490991815,2cyclictest8759-21kworker/u8:220:16:051
16490991815,2cyclictest28905-21kworker/u8:120:04:011
16490991815,2cyclictest24129-21kworker/u8:222:21:041
16490991815,2cyclictest22200-21kworker/u8:119:26:041
16490991815,2cyclictest19337-21kworker/u8:100:24:051
16490991815,2cyclictest19337-21kworker/u8:100:24:051
16490991815,2cyclictest16367-21kworker/u8:019:22:041
16490991815,2cyclictest16367-21kworker/u8:019:17:041
16490991815,2cyclictest15312-21kworker/u8:022:33:041
16490991815,2cyclictest12158-21kworker/u8:223:32:041
16490991813,4cyclictest7376-21kworker/u8:122:28:051
16490991813,4cyclictest24352-21kworker/u8:222:01:041
272182170,0sleep10-21swapper/121:37:251
263962170,1sleep30-21swapper/323:51:593
229582170,1sleep30-21swapper/300:22:133
229582170,1sleep30-21swapper/300:22:133
1649299173,12cyclictest30512-21modprobe00:30:163
1649299173,12cyclictest30512-21modprobe00:30:163
16492991713,2cyclictest7046-21sh23:31:383
1649199172,9cyclictest16672-21ssh22:35:002
1649199172,2cyclictest4077-21ssh00:36:012
16491991714,2cyclictest25924-21kworker/u8:023:22:042
16491991712,3cyclictest0-21swapper/200:00:182
1649099172,2cyclictest0-21swapper/123:05:041
16490991714,2cyclictest6110-21kworker/u8:020:45:041
16490991714,2cyclictest6110-21kworker/u8:020:36:041
16490991714,2cyclictest28905-21kworker/u8:119:56:041
16490991714,2cyclictest28905-21kworker/u8:119:41:041
16490991714,2cyclictest24352-21kworker/u8:221:58:041
16490991714,2cyclictest24352-21kworker/u8:221:51:041
16490991714,2cyclictest24129-21kworker/u8:222:18:041
16490991714,2cyclictest18814-21kworker/u8:121:06:041
16490991714,2cyclictest1624-21kworker/u8:221:17:051
16490991714,2cyclictest14299-21kworker/u8:119:11:041
16490991714,2cyclictest14299-21kworker/u8:119:11:041
16490991714,2cyclictest11163-21kworker/u8:123:36:051
1649099170,2cyclictest0-21swapper/123:47:041
1649099170,2cyclictest0-21swapper/100:26:041
1649099170,12cyclictest0-21swapper/121:27:041
1648999171,14cyclictest1139-21apt-get21:45:060
16492991611,3cyclictest26484-21ls00:25:283
1649199161,3cyclictest30163-21apt-get22:15:162
16491991613,2cyclictest9099-21kworker/u8:200:14:042
1649199161,13cyclictest16091-21ssh22:00:132
1649199160,2cyclictest0-21swapper/221:31:042
16490991613,2cyclictest9099-21kworker/u8:200:37:051
16490991613,2cyclictest7376-21kworker/u8:122:36:051
16490991613,2cyclictest6110-21kworker/u8:020:31:041
16490991613,2cyclictest6110-21kworker/u8:020:24:041
16490991613,2cyclictest31296-21kworker/u8:123:24:041
16490991613,2cyclictest2921-21kworker/u8:100:12:041
16490991613,2cyclictest28905-21kworker/u8:119:50:051
16490991613,2cyclictest26258-21kworker/u8:219:39:041
16490991613,2cyclictest24352-21kworker/u8:221:46:041
16490991613,2cyclictest20739-21kworker/u8:222:41:041
16490991613,2cyclictest1900-21kworker/u8:122:53:041
16490991613,2cyclictest18814-21kworker/u8:120:59:041
16490991613,2cyclictest12799-21kworker/u8:120:28:011
1649099160,2cyclictest0-21swapper/122:45:041
1649099160,11cyclictest0-21swapper/122:05:051
1649299151,12cyclictest14147-21ssh21:57:503
1649299150,12cyclictest0-21swapper/323:14:093
1649199151,2cyclictest0-21swapper/222:07:042
16491991512,2cyclictest18814-21kworker/u8:120:47:042
16491991510,3cyclictest1139-21apt-get21:45:052
16491991510,3cyclictest0-21swapper/222:13:042
16491991510,3cyclictest0-21swapper/222:13:032
1649199150,2cyclictest0-21swapper/222:45:232
16490991512,2cyclictest6110-21kworker/u8:020:13:041
16490991512,2cyclictest28905-21kworker/u8:120:08:041
16490991512,2cyclictest28826-21kworker/u8:000:05:041
16490991512,2cyclictest18814-21kworker/u8:120:54:011
1649099150,7cyclictest0-21swapper/123:55:121
1649099150,2cyclictest0-21swapper/121:23:041
16489991511,2cyclictest0-21swapper/000:20:060
16489991511,2cyclictest0-21swapper/000:20:060
1648999150,13cyclictest0-21swapper/023:11:260
1649299149,3cyclictest17506-21hddtemp_smartct22:35:173
1649299149,3cyclictest0-21swapper/321:31:003
16492991411,2cyclictest32602-21diskmemload22:14:473
16492991411,2cyclictest32602-21diskmemload22:14:473
1649299140,2cyclictest0-21swapper/322:55:273
1649299140,13cyclictest0-21swapper/323:20:573
1649299140,12cyclictest20504-21timerwakeupswit23:45:273
1649299140,12cyclictest12291-21ssh23:04:063
1649299140,12cyclictest0-21swapper/323:35:163
1649299140,12cyclictest0-21swapper/321:28:413
1649299140,12cyclictest0-21swapper/300:01:493
1649199142,7cyclictest15547-21ssh22:00:002
16491991411,2cyclictest19013-21kworker/u8:223:50:042
16491991411,2cyclictest18814-21kworker/u8:121:10:042
16491991411,2cyclictest0-21swapper/223:29:272
1649199140,3cyclictest1925-21latency_hist00:35:002
1649199140,3cyclictest1925-21latency_hist00:35:002
1649199140,2cyclictest0-21swapper/223:00:182
1649199140,2cyclictest0-21swapper/221:36:072
1649199140,2cyclictest0-21swapper/200:05:272
1649199140,12cyclictest0-21swapper/223:10:122
16490991411,2cyclictest19021-21kworker/u8:122:10:041
16490991411,2cyclictest19021-21kworker/u8:122:10:041
16490991411,2cyclictest18814-21kworker/u8:121:02:041
16490991411,2cyclictest17874-21kworker/u8:123:16:041
16490991411,2cyclictest0-21swapper/122:55:181
16490991411,2cyclictest0-21swapper/100:30:081
16490991411,2cyclictest0-21swapper/100:30:081
1649099140,10cyclictest0-21swapper/123:01:051
1649099140,10cyclictest0-21swapper/100:18:011
1649099140,10cyclictest0-21swapper/100:18:011
1648999145,3cyclictest278-21jbd2/sda1-823:25:000
1648999142,7cyclictest32602-21diskmemload21:55:000
1648999140,13cyclictest0-21swapper/000:15:170
1648999140,13cyclictest0-21swapper/000:15:160
1648999140,12cyclictest0-21swapper/023:25:160
1648999140,12cyclictest0-21swapper/022:57:260
1648999140,12cyclictest0-21swapper/020:55:210
1649299138,3cyclictest5010-21snmpd19:15:053
1649299138,3cyclictest17809-21ssh00:16:403
1649299138,3cyclictest17809-21ssh00:16:403
16492991310,2cyclictest1022-21ssh22:18:183
16492991310,2cyclictest0-21swapper/323:57:113
16492991310,2cyclictest0-21swapper/323:40:563
16492991310,2cyclictest0-21swapper/323:15:173
16492991310,2cyclictest0-21swapper/322:30:073
16492991310,2cyclictest0-21swapper/321:42:483
16492991310,2cyclictest0-21swapper/321:36:273
16492991310,2cyclictest0-21swapper/321:20:143
16492991310,2cyclictest0-21swapper/321:15:423
16492991310,2cyclictest0-21swapper/320:05:133
1649299130,3cyclictest0-21swapper/322:48:143
1649299130,2cyclictest32602-21diskmemload22:40:363
1649299130,2cyclictest32248-21ssh23:25:133
1649299130,2cyclictest14294-21ssh23:05:233
1649299130,2cyclictest0-21swapper/322:20:153
1649299130,2cyclictest0-21swapper/321:50:283
1649299130,2cyclictest0-21swapper/321:14:483
1649299130,12cyclictest0-21swapper/322:25:573
1649299130,12cyclictest0-21swapper/322:00:173
1649299130,12cyclictest0-21swapper/300:14:373
1649299130,11cyclictest0-21swapper/300:05:493
1649199138,3cyclictest16697-21munin-node2
1649199132,6cyclictest0-21swapper/219:20:002
1649199132,6cyclictest0-21swapper/200:20:002
1649199132,6cyclictest0-21swapper/200:20:002
16491991310,2cyclictest4039-21kworker/u8:023:35:052
16491991310,2cyclictest22200-21kworker/u8:119:27:052
16491991310,2cyclictest12651-21ssh21:21:572
16491991310,2cyclictest0-21swapper/223:49:072
16491991310,2cyclictest0-21swapper/222:28:152
1649199130,8cyclictest0-21swapper/219:13:042
1649199130,8cyclictest0-21swapper/219:13:042
1649199130,2cyclictest0-21swapper/223:08:432
1649199130,2cyclictest0-21swapper/221:50:472
1649199130,2cyclictest0-21swapper/220:10:132
1649199130,11cyclictest32602-21diskmemload23:41:442
16490991310,2cyclictest32602-21diskmemload23:50:181
1649099130,3cyclictest0-21swapper/121:12:041
1649099130,2cyclictest0-21swapper/121:34:041
1649099130,12cyclictest0-21swapper/123:10:001
1648999131,9cyclictest32602-21diskmemload23:05:520
16489991310,2cyclictest0-21swapper/000:25:360
1648999130,12cyclictest0-21swapper/023:30:560
1648999130,12cyclictest0-21swapper/022:20:170
1648999130,11cyclictest3949-21ssh00:35:500
1648999130,11cyclictest0-21swapper/021:20:270
65012120,1sleep00-21swapper/021:15:240
304152120,2sleep00-21swapper/022:49:200
1649299129,2cyclictest0-21swapper/319:20:353
1649299129,2cyclictest0-21swapper/319:10:363
1649299129,2cyclictest0-21swapper/319:10:363
1649299127,3cyclictest0-21swapper/322:05:273
1649299120,2cyclictest0-21swapper/320:45:163
1649299120,2cyclictest0-21swapper/319:45:183
1649299120,11cyclictest0-21swapper/320:20:123
1649299120,11cyclictest0-21swapper/319:50:193
1649299120,11cyclictest0-21swapper/319:30:263
1649299120,11cyclictest0-21swapper/319:25:183
1649199129,2cyclictest7376-21kworker/u8:122:40:042
1649199129,2cyclictest22331-21kworker/u8:323:56:052
1649199129,2cyclictest0-21swapper/222:50:072
1649199129,2cyclictest0-21swapper/221:25:212
1649199129,2cyclictest0-21swapper/220:40:202
1649199129,2cyclictest0-21swapper/220:35:192
1649199127,3cyclictest30586-21cron21:05:002
1649199122,6cyclictest0-21swapper/219:40:002
1649199120,2cyclictest0-21swapper/219:20:162
1649199120,2cyclictest0-21swapper/200:28:172
1649199120,11cyclictest0-21swapper/221:40:272
1649199120,11cyclictest0-21swapper/220:15:142
1649199120,11cyclictest0-21swapper/219:55:162
1649099129,2cyclictest0-21swapper/123:26:551
1648999129,2cyclictest32602-21diskmemload23:54:570
1648999129,2cyclictest0-21swapper/023:19:360
1648999129,2cyclictest0-21swapper/022:32:060
1648999129,2cyclictest0-21swapper/022:26:070
1648999129,2cyclictest0-21swapper/019:30:160
1648999129,2cyclictest0-21swapper/000:30:560
1648999129,2cyclictest0-21swapper/000:30:560
1648999127,3cyclictest14919-21cut20:25:150
1648999127,3cyclictest0-21swapper/021:05:180
1648999122,7cyclictest0-21swapper/000:08:580
1648999122,6cyclictest0-21swapper/020:10:170
1648999122,6cyclictest0-21swapper/020:00:000
1648999122,5cyclictest0-21swapper/022:40:010
1648999120,2cyclictest0-21swapper/000:04:260
1648999120,11cyclictest0-21swapper/022:40:360
1648999120,11cyclictest0-21swapper/022:17:560
1648999120,11cyclictest0-21swapper/021:29:140
1648999120,11cyclictest0-21swapper/019:45:180
1648999120,11cyclictest0-21swapper/019:40:370
9772110,1sleep00-21swapper/019:50:180
257602110,2sleep20-21swapper/220:50:232
1649299118,2cyclictest0-21swapper/320:40:153
1649299118,2cyclictest0-21swapper/320:38:003
1649299118,2cyclictest0-21swapper/320:25:303
1649299118,2cyclictest0-21swapper/320:15:153
1649199113,6cyclictest0-21swapper/219:45:262
1649199110,10cyclictest0-21swapper/220:07:572
1648999118,2cyclictest31795-21ssh21:42:070
1648999118,2cyclictest0-21swapper/021:57:070
1648999118,2cyclictest0-21swapper/019:39:360
1648999116,3cyclictest30203-21grep21:00:280
1648999111,8cyclictest0-21swapper/023:40:200
1648999111,5cyclictest0-21swapper/020:05:010
1648999111,5cyclictest0-21swapper/019:10:000
1648999111,5cyclictest0-21swapper/019:10:000
1648999110,10cyclictest0-21swapper/023:36:360
1648999110,10cyclictest0-21swapper/022:12:560
1648999110,10cyclictest0-21swapper/022:12:560
1648999110,10cyclictest0-21swapper/022:02:160
1648999110,10cyclictest0-21swapper/021:30:270
283932100,1sleep30-21swapper/319:37:553
1649299107,2cyclictest0-21swapper/319:57:573
1649299102,7cyclictest0-21swapper/320:30:013
1649299100,6cyclictest0-21swapper/321:05:003
1649199107,2cyclictest26258-21kworker/u8:219:33:042
1649199107,2cyclictest12799-21kworker/u8:120:27:042
1649199104,3cyclictest1685-21ls19:50:282
1648999107,2cyclictest0-21swapper/019:25:160
1648999107,2cyclictest0-21swapper/019:20:140
1648999105,3cyclictest648-21cron00:00:000
1648999105,3cyclictest23357-21cut20:45:170
1648999105,3cyclictest21535-21needreboot22:05:200
1648999102,6cyclictest0-21swapper/023:00:510
1648999100,9cyclictest0-21swapper/000:10:560
1648999100,6cyclictest32602-21diskmemload21:35:270
164929994,3cyclictest30934-21head21:05:143
164929994,3cyclictest27060-21grep20:55:173
164919994,3cyclictest27043-21tr20:55:172
164919994,3cyclictest14463-21latency_hist20:25:002
164919991,6cyclictest0-21swapper/219:40:102
164899995,2cyclictest0-21swapper/020:30:130
164899994,3cyclictest12812-21aten2.4_r3power20:20:130
164899992,6cyclictest0-21swapper/019:15:060
164899991,6cyclictest0-21swapper/020:40:130
164899990,8cyclictest32602-21diskmemload22:50:590
164929983,3cyclictest8752-21aten2.4-expect20:10:143
164929983,3cyclictest4638-21egrep20:00:153
164919981,4cyclictest0-21swapper/220:00:122
164899984,2cyclictest5129-21runrttasks20:15:050
164899983,3cyclictest4708-21tr20:00:140
164899983,3cyclictest24778-21aten2.4_r3power20:50:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional