You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-12 - 19:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System (updated Fri Dec 12, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2811382212264,54sleep30-21swapper/319:06:213
281134128956,10sleep20-21swapper/219:05:482
281165628554,10sleep10-21swapper/119:09:481
281162227552,6sleep00-21swapper/019:09:220
281180899279,13cyclictest0-21swapper/322:35:123
2811808992519,3cyclictest0-21swapper/321:10:283
2811808992420,2cyclictest0-21swapper/322:10:193
2811808992414,7cyclictest3222247-21turbostat00:05:003
32201092231,3sleep20-21swapper/200:01:462
30356312231,3sleep20-21swapper/222:07:292
281180899237,11cyclictest2997499-21chrt21:45:003
2811808992319,2cyclictest0-21swapper/320:05:003
32657132221,3sleep00-21swapper/000:30:270
281180899227,10cyclictest3003579-21taskset21:47:043
281180899226,13cyclictest0-21swapper/323:43:053
281180899225,11cyclictest3172661-21taskset23:33:523
281180899224,12cyclictest3141695-21taskset23:15:123
2811808992216,3cyclictest0-21swapper/323:24:523
2811808992215,2cyclictest0-21swapper/300:07:583
281180899221,15cyclictest2877816-21chrt20:10:123
281180899221,10cyclictest2980733-21chrt21:33:493
281180899220,15cyclictest3205214-21taskset23:53:523
281180899214,13cyclictest0-21swapper/321:15:213
281180899212,16cyclictest2817575-21chrt19:15:023
281180899212,12cyclictest3065826-21taskset22:25:253
2811808992118,1cyclictest0-21swapper/322:06:373
2811808992118,1cyclictest0-21swapper/300:32:323
2811808992117,2cyclictest0-21swapper/323:11:013
2811808992113,6cyclictest15950irq/132-ahci[0000:00:17.0]21:00:193
281180899211,15cyclictest0-21swapper/323:55:143
281180899211,12cyclictest31299882sleep323:05:223
30690222201,4sleep22941487-21diskmemload22:30:022
29801792201,6sleep10-21swapper/121:32:521
281180899208,1cyclictest0-21swapper/319:35:143
281180899207,9cyclictest3089699-21chrt22:40:203
281180899206,12cyclictest0-21swapper/323:46:583
281180899204,12cyclictest0-21swapper/320:50:173
281180899202,12cyclictest2981809-21taskset21:35:103
281180899202,11cyclictest2898852-21chrt20:28:363
2811808992017,1cyclictest0-21swapper/322:56:433
2811808992017,1cyclictest0-21swapper/322:52:533
2811808992016,2cyclictest0-21swapper/322:24:353
2811808992016,2cyclictest0-21swapper/321:05:243
2811808992016,2cyclictest0-21swapper/320:35:003
2811808992015,4cyclictest0-21swapper/323:00:213
2811808992015,3cyclictest0-21swapper/322:17:323
2811808992015,3cyclictest0-21swapper/300:38:303
281180899201,15cyclictest3230600-21chrt00:10:123
281180899201,14cyclictest2829968-21chrt19:25:123
281180899201,12cyclictest3258459-21taskset00:25:313
281180899194,13cyclictest0-21swapper/300:22:233
2811808991916,2cyclictest0-21swapper/321:22:423
2811808991916,1cyclictest0-21swapper/322:31:483
2811808991916,1cyclictest0-21swapper/319:50:133
2811808991915,2cyclictest0-21swapper/322:03:013
2811808991914,2cyclictest0-21swapper/321:52:283
2811808991912,4cyclictest31629072sleep323:26:033
281180899191,15cyclictest0-21swapper/323:35:183
281180899191,14cyclictest0-21swapper/300:16:423
32226102183,9sleep13222630-21apt-get00:05:141
31653402181,3sleep23165354-21ssh23:29:592
30507892181,8sleep13050795-21ssh22:16:061
281180899185,1cyclictest0-21swapper/319:20:243
281180899184,10cyclictest3013601-21chrt21:55:103
281180899182,10cyclictest2850423-21taskset19:40:313
2811808991815,1cyclictest0-21swapper/320:20:293
2811808991815,1cyclictest0-21swapper/320:16:453
2811808991814,2cyclictest0-21swapper/320:35:033
281180899181,13cyclictest2965925-21chrt21:25:073
281180899181,13cyclictest0-21swapper/319:45:113
281180899181,11cyclictest3100030-21taskset22:47:463
281180899180,14cyclictest0-21swapper/320:40:113
281180899180,13cyclictest0-21swapper/319:55:123
32118832171,4sleep12941487-21diskmemload23:56:361
30847902172,9sleep0802-21snmpd22:39:500
29574492171,3sleep20-21swapper/221:19:502
29417922171,4sleep22941487-21diskmemload21:10:082
28873842171,10sleep12887432-21ntpq20:15:201
281180899176,8cyclictest2837416-21chrt19:30:173
281180899176,2cyclictest0-21swapper/320:46:533
281180899171,4cyclictest0-21swapper/320:05:133
2811808991713,2cyclictest2816650-21taskset19:10:293
32674942161,3sleep12941487-21diskmemload00:32:351
31985212161,9sleep23198736-21hddtemp_smartct23:50:152
31723432161,10sleep03172356-21ssh23:33:220
31571632161,9sleep12941487-21diskmemload23:24:371
31389702160,10sleep12941487-21diskmemload23:11:141
30295312161,9sleep02941487-21diskmemload22:05:060
30007942161,8sleep13001701-21memory21:45:191
29554982161,2sleep10-21swapper/121:16:411
281180899168,4cyclictest2926153-21taskset20:55:133
31960102151,9sleep12941487-21diskmemload23:47:061
31236542151,9sleep22941487-21diskmemload23:02:092
31139352151,9sleep03113988-21gcc22:55:220
30758332151,7sleep12941487-21diskmemload22:33:091
30659962151,4sleep23066245-21users22:25:292
30656362151,10sleep03065697-21ssh22:25:230
29957762151,3sleep02995783-21ssh21:42:210
29744672151,9sleep22974699-21cat21:30:172
29645352152,5sleep02964547-21pool-udisksd21:23:210
29560202151,9sleep02956035-21ssh21:17:360
32001352141,9sleep13201898-21cut23:50:181
31318672141,8sleep03131881-21ssh23:07:330
31164482141,7sleep22941487-21diskmemload22:58:162
31017402140,9sleep23101751-21ssh22:50:062
30912772141,3sleep13091287-21ssh22:41:511
30361632143,7sleep12941487-21diskmemload22:08:201
30189582141,3sleep22941487-21diskmemload21:56:142
29889822141,6sleep00-21swapper/021:39:210
29782252141,7sleep02941487-21diskmemload21:30:270
28355142141,8sleep02835725-21smartctl19:30:150
32756072131,5sleep00-21swapper/000:38:560
32621162131,8sleep22941487-21diskmemload00:30:202
32524592131,8sleep22941487-21diskmemload00:23:392
31957722131,8sleep23195778-21sh23:46:432
31492682130,1sleep10-21swapper/123:19:481
31478302131,8sleep23147842-21rm23:17:322
31007292131,6sleep20-21swapper/222:48:542
30996752130,6sleep00-21swapper/022:47:110
28415252131,8sleep12843230-21cut19:35:171
32591372121,6sleep12941487-21diskmemload00:26:381
32362202120,1sleep00-21swapper/000:11:580
31154462121,5sleep12941487-21diskmemload22:56:391
31067052121,5sleep12941487-21diskmemload22:50:371
30762342121,7sleep03076247-21ssh22:33:490
30596852121,5sleep20-21swapper/222:22:442
30136532121,4sleep00-21swapper/021:55:160
29634382121,3sleep10-21swapper/121:21:341
29141372121,6sleep12914189-21kworker/u16:320:40:211
28892012121,6sleep12890213-21kthreadcore20:20:171
28820902121,7sleep02882193-21timerandwakeup20:10:260
28243222121,7sleep02824422-21latency_hist19:20:010
32597192111,3sleep00-21swapper/000:27:370
32536272111,5sleep23253651-21apt-get00:25:152
32525562111,4sleep12941487-21diskmemload00:23:501
32437502111,3sleep00-21swapper/000:16:470
32433072110,1sleep20-21swapper/200:16:012
32382842111,5sleep13238545-21df00:15:161
32271172111,4sleep22941487-21diskmemload00:05:272
32205022111,4sleep12941487-21diskmemload00:02:211
32130522110,8sleep00-21swapper/023:58:310
31963382111,4sleep02941487-21diskmemload23:47:370
31849962111,5sleep23186027-21memory23:40:172
31807032111,4sleep12941487-21diskmemload23:38:531
31732392111,4sleep12941487-21diskmemload23:34:471
31703072111,3sleep20-21swapper/223:30:272
31570012111,4sleep22941487-21diskmemload23:24:222
31406692111,4sleep22941487-21diskmemload23:13:592
31317152110,7sleep23131726-21ssh23:07:182
31257762111,3sleep13125800-21apt-get23:05:121
31178002111,4sleep12941487-21diskmemload23:00:071
31177912111,4sleep02941487-21diskmemload23:00:060
30991162111,4sleep12941487-21diskmemload22:46:201
30904432111,4sleep02941487-21diskmemload22:40:300
30846312111,6sleep23084643-21sh22:39:322
30827352111,4sleep12941487-21diskmemload22:36:221
30427702111,3sleep00-21swapper/022:11:000
30276112111,3sleep00-21swapper/022:02:220
30272782111,4sleep22941487-21diskmemload22:01:482
30272242110,1sleep10-21swapper/122:01:431
30047682111,3sleep20-21swapper/221:49:092
29969552111,2sleep20-21swapper/221:44:162
29950422111,4sleep12941487-21diskmemload21:41:101
29875942111,4sleep12941487-21diskmemload21:36:571
29867692111,3sleep20-21swapper/221:35:382
29714912111,4sleep02941487-21diskmemload21:26:420
29584052111,3sleep20-21swapper/221:20:182
29483562111,4sleep12941487-21diskmemload21:13:001
29404702111,5sleep02940505-21ntpq21:05:250
28902162111,6sleep22891996-21kthreadcore20:20:182
28829302111,4sleep22882793-21apt-get20:15:012
28765912111,5sleep12876648-21ntp_states20:05:221
28730942110,3sleep02874284-21kthreadcore20:05:170
28619172111,6sleep22862178-21/usr/sbin/munin19:55:132
28392502111,5sleep22839300-21ntpq19:30:212
32741772101,3sleep12941487-21diskmemload00:35:521
32695032101,4sleep23269639-21perl00:35:172
32306232101,3sleep10-21swapper/100:10:151
32305932101,3sleep20-21swapper/200:10:122
32291332101,2sleep00-21swapper/000:08:050
32200272101,3sleep02941487-21diskmemload00:01:360
32126742101,3sleep20-21swapper/223:57:572
32029822100,1sleep00-21swapper/023:50:290
31787782101,2sleep00-21swapper/023:35:460
31784092101,3sleep20-21swapper/223:35:282
31647182101,3sleep12941487-21diskmemload23:28:581
31560912101,3sleep00-21swapper/023:22:510
31475872101,2sleep00-21swapper/023:17:110
31336762101,3sleep00-21swapper/023:10:120
31018012101,3sleep02941487-21diskmemload22:50:120
30929072101,3sleep20-21swapper/222:44:212
30669202101,2sleep10-21swapper/122:26:361
30585802101,3sleep00-21swapper/022:20:540
30508642101,4sleep02941487-21diskmemload22:16:130
30507652101,2sleep20-21swapper/222:16:032
30449112100,3sleep10-21swapper/122:14:291
30389662101,5sleep23040633-21kthreadcore22:10:172
30136242101,4sleep12941487-21diskmemload21:55:131
30118192101,3sleep20-21swapper/221:52:382
30106162101,4sleep02941487-21diskmemload21:50:390
29659142101,3sleep22941487-21diskmemload21:25:072
29659052101,3sleep12941487-21diskmemload21:25:061
29470612101,3sleep00-21swapper/021:10:590
29300052100,1sleep00-21swapper/020:55:230
29264812101,4sleep22928086-21cut20:55:172
29140882101,4sleep02914135-21ntpq20:40:200
28942212101,4sleep02894430-21hddtemp_smartct20:25:170
28936232100,1sleep00-21swapper/020:24:290
28838742101,5sleep02884577-21cut20:15:150
28822032101,4sleep12882410-21users20:10:271
28706622101,5sleep12871206-21strings20:00:171
28624602101,5sleep12863380-21cut19:55:161
28563332101,4sleep22856352-21apt-get19:50:122
28233052101,4sleep12823351-21ntpq19:15:211
28171762101,4sleep22817569-21ps19:15:022
3250880290,2sleep00-21swapper/000:20:360
3188189290,2sleep00-21swapper/023:42:360
3187796290,3sleep10-21swapper/123:41:561
3161951290,3sleep00-21swapper/023:25:240
3058829291,3sleep13058839-21ssh22:21:191
3005597291,2sleep10-21swapper/121:50:041
3005120290,3sleep00-21swapper/021:49:430
2940507291,3sleep12940554-21ntpq21:05:261
2931173291,2sleep20-21swapper/221:00:142
2929849291,4sleep12929905-21ld20:55:201
2924961291,4sleep22925229-21sed20:50:282
2924665291,3sleep12924721-21ntpq20:50:231
2908837291,3sleep12908891-21ntpq20:35:241
2893930291,3sleep12893949-21apt-get20:25:141
2877794291,3sleep20-21swapper/220:10:122
2872812291,4sleep22873014-21gltestperf20:05:152
2849792291,4sleep12849843-21ld19:40:221
2841041292,3sleep02841521-21kthreadcore19:35:160
2840235291,4sleep12840276-21systemd-run19:35:001
2824607291,3sleep12824626-21apt-get19:20:141
2812249291,4sleep02812319-21fschecks_time19:10:150
2811969291,2sleep20-21swapper/219:10:132
2933789281,3sleep02935047-21/usr/sbin/munin21:00:200
2908797281,2sleep02908835-21ntpq20:35:230
2908163281,3sleep22904800-21/usr/sbin/munin20:35:212
2849844280,1sleep20-21swapper/219:40:232
2829946280,3sleep20-21swapper/219:25:122
2941202270,1sleep20-21swapper/221:06:312
2920146271,3sleep2802-21snmpd20:47:162
2914832270,1sleep20-21swapper/220:41:432
2903954270,1sleep10-21swapper/120:31:121
2898754270,1sleep20-21swapper/220:27:242
2858281271,2sleep02858397-21kthreadcore19:50:190
2851059271,2sleep20-21swapper/219:45:122
2845767270,2sleep00-21swapper/019:40:130
2816774270,1sleep10-21swapper/119:12:041
2925559260,1sleep00-21swapper/020:54:190
2920031260,1sleep00-21swapper/020:45:450
2899794261,2sleep20-21swapper/220:30:162
2868860261,2sleep00-21swapper/020:00:160
2867223260,3sleep2421rcuc/220:00:102
2851037260,2sleep10-21swapper/119:45:101
2825430261,2sleep20-21swapper/219:20:192
2935947250,3sleep10-21swapper/121:01:461
2915868250,2sleep10-21swapper/120:45:181
2865846250,1sleep00-21swapper/019:55:200
2851328250,1sleep00-21swapper/019:45:140
2843232250,1sleep20-21swapper/219:35:182
2830222250,1sleep00-21swapper/019:25:140
2899513240,1sleep00-21swapper/020:30:140
2856624240,1sleep10-21swapper/119:50:161
2830247240,1sleep10-21swapper/119:25:151
2824908240,1sleep00-21swapper/019:20:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional