You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-20 - 20:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Nov 20, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
309262239162,24sleep10-21swapper/107:07:221
309792237163,23sleep20-21swapper/207:08:022
308392237164,21sleep30-21swapper/307:06:303
306932225195,20sleep00-21swapper/007:05:280
584221640,5sleep13123099cyclictest11:03:421
3123799105100,2cyclictest20872chrt12:08:132
249362650,6sleep121-21rcuc/109:20:131
31237996057,2cyclictest0-21swapper/207:21:302
54782570,5sleep121-21rcuc/112:15:181
150162570,2sleep20-21swapper/210:10:192
157192550,5sleep011-21rcuc/011:25:200
157192550,5sleep011-21rcuc/011:25:200
307222540,4sleep030727-21cpuspeed_turbos08:20:150
263092530,4sleep230-21rcuc/208:10:122
131392520,2sleep00-21swapper/010:05:200
188442510,1sleep20-21swapper/209:05:162
31237994533,2cyclictest21780-21kworker/2:008:40:232
3124299416,7cyclictest780-21rngd11:51:233
31242994024,4cyclictest0-21swapper/308:40:243
3124299376,8cyclictest780-21rngd09:59:433
3124299375,5cyclictest780-21rngd09:17:293
3124299375,5cyclictest780-21rngd09:17:293
3124299357,8cyclictest780-21rngd07:35:043
3124299357,16cyclictest780-21rngd11:32:193
31226993510,12cyclictest8846-21perf12:25:010
31242993416,9cyclictest0-21swapper/309:50:253
3122699348,12cyclictest3730-21munin-run11:00:020
3124299335,5cyclictest780-21rngd11:26:273
3124299335,5cyclictest780-21rngd11:26:273
3123099338,17cyclictest24865-21ntp_kernel_pll_11:45:241
3124299325,4cyclictest780-21rngd08:38:443
3124299324,5cyclictest780-21rngd10:05:453
31237993221,2cyclictest8177-21kworker/2:111:38:472
3124299316,8cyclictest780-21rngd08:46:193
3124299314,4cyclictest780-21rngd08:26:123
3123799312,20cyclictest21780-21kworker/2:008:37:272
31237993121,5cyclictest21780-21kworker/2:010:39:592
31237993121,2cyclictest21780-21kworker/2:010:23:212
31237993111,5cyclictest8177-21kworker/2:112:10:192
3122699316,12cyclictest3933-21munin-run07:20:000
3122699313,13cyclictest28156-21idleruntime-cro08:15:010
31237993027,2cyclictest21780-21kworker/2:009:50:252
31237993021,3cyclictest8177-21kworker/2:112:39:352
3123799302,12cyclictest8177-21kworker/2:112:03:452
31237993020,2cyclictest21780-21kworker/2:010:44:192
31237993019,2cyclictest8177-21kworker/2:111:57:212
31226993011,16cyclictest15601-21awk09:00:010
3124299294,8cyclictest780-21rngd10:59:433
31242992914,8cyclictest0-21swapper/311:37:513
3122699291,13cyclictest32413-21perf12:05:020
3124299284,8cyclictest780-21rngd11:00:553
3124299284,7cyclictest780-21rngd07:49:443
31237992818,2cyclictest8177-21kworker/2:112:16:072
31237992817,3cyclictest21780-21kworker/2:009:20:072
31237992815,2cyclictest21780-21kworker/2:009:38:232
3123099289,10cyclictest32604-21cut08:25:011
3123099285,15cyclictest0-21swapper/110:45:161
3124299277,8cyclictest780-21rngd11:11:133
3124299277,7cyclictest780-21rngd09:28:573
3124299275,7cyclictest780-21rngd08:24:263
3124299274,7cyclictest780-21rngd10:43:073
3123099271,18cyclictest5066-21chrt09:47:301
3122699279,15cyclictest10371-21awk11:15:010
3122699272,12cyclictest11207-21perf08:50:020
76702260,3sleep011-21rcuc/011:05:250
3124299267,6cyclictest780-21rngd11:56:093
3124299266,7cyclictest780-21rngd12:36:313
3124299265,5cyclictest780-21rngd11:21:153
3124299261,3cyclictest0-21swapper/312:31:463
3124299261,3cyclictest0-21swapper/312:31:463
3124299260,7cyclictest0-21swapper/308:33:543
31237992620,2cyclictest21780-21kworker/2:009:13:552
31237992617,2cyclictest21780-21kworker/2:009:55:352
31237992615,2cyclictest8177-21kworker/2:111:31:232
3122699261,12cyclictest3092-21basename09:45:020
268212261,21sleep126823-21acpi09:25:121
3124299256,5cyclictest86550irq/126-enp1s0-10:54:553
3124299255,7cyclictest780-21rngd11:45:213
3124299254,8cyclictest780-21rngd10:46:193
31237992515,2cyclictest8177-21kworker/2:111:51:472
31237992514,2cyclictest31650-21kworker/2:207:38:052
3123799251,21cyclictest21780-21kworker/2:010:30:212
3123099252,9cyclictest784-21NetworkManager09:00:131
31230992516,3cyclictest0-21swapper/110:28:131
31230992514,3cyclictest12491-21tune2fs10:05:151
3124299247,5cyclictest86850irq/129-enp1s0-10:18:113
3124299246,4cyclictest780-21rngd09:49:513
3124299244,7cyclictest780-21rngd08:54:393
3124299243,6cyclictest0-21swapper/307:56:243
3124299242,7cyclictest780-21rngd11:19:313
31237992413,2cyclictest21780-21kworker/2:009:30:132
31237992410,2cyclictest21780-21kworker/2:011:16:432
3123099241,8cyclictest0-21swapper/111:17:141
31230992415,3cyclictest0-21swapper/110:15:581
3123099241,17cyclictest0-21swapper/111:22:081
3123099241,17cyclictest0-21swapper/109:11:451
3123099241,17cyclictest0-21swapper/107:41:081
31230992411,6cyclictest10882-21kworker/u8:212:28:501
3123099241,16cyclictest0-21swapper/110:50:211
3123099240,17cyclictest0-21swapper/109:40:251
3123099240,17cyclictest0-21swapper/108:25:111
3123099240,16cyclictest0-21swapper/112:05:421
318842230,4sleep13123099cyclictest09:35:141
316432230,2sleep081ktimersoftd/009:35:130
3124299235,5cyclictest0-21swapper/307:24:083
3124299235,3cyclictest86650irq/127-enp1s0-12:05:243
3124299234,5cyclictest0-21swapper/307:32:283
3124299233,7cyclictest780-21rngd12:22:103
3124299231,3cyclictest0-21swapper/310:20:063
3123799239,8cyclictest21780-21kworker/2:010:59:212
3123799239,3cyclictest8177-21kworker/2:111:29:272
3123799239,3cyclictest8177-21kworker/2:111:29:272
3123799237,4cyclictest19360-21awk07:55:002
3123799232,2cyclictest31650-21kworker/2:207:20:022
3123799231,6cyclictest0-21swapper/212:33:242
3123799231,6cyclictest0-21swapper/212:33:242
3123799231,5cyclictest0-21swapper/210:48:242
3123799231,5cyclictest0-21swapper/210:03:462
3123799231,4cyclictest0-21swapper/211:49:012
31237992312,3cyclictest21780-21kworker/2:008:20:312
31237992311,3cyclictest21780-21kworker/2:008:28:472
3123099232,16cyclictest0-21swapper/109:19:181
3123099232,16cyclictest0-21swapper/109:19:181
31230992315,3cyclictest0-21swapper/108:51:001
31230992315,3cyclictest0-21swapper/108:34:341
31230992314,3cyclictest0-21swapper/107:17:011
3123099231,16cyclictest0-21swapper/111:55:041
3123099231,15cyclictest20004-21diskmemload12:13:321
31230992311,3cyclictest28156-21sh08:15:001
3122699231,6cyclictest0-21swapper/012:33:500
3122699231,6cyclictest0-21swapper/012:33:500
321402220,4sleep07-21ksoftirqd/007:10:170
3124299224,4cyclictest780-21rngd07:18:183
3124299223,3cyclictest780-21rngd12:13:363
3124299221,3cyclictest0-21swapper/311:41:593
31242992211,4cyclictest0-21swapper/310:25:153
3123799229,3cyclictest86750irq/128-enp1s0-08:50:352
3123799228,8cyclictest86550irq/126-enp1s0-09:29:112
3123799228,3cyclictest21780-21kworker/2:011:06:252
3123799227,3cyclictest21780-21kworker/2:011:00:152
3123799225,3cyclictest86850irq/129-enp1s0-08:45:252
31237992213,2cyclictest21780-21kworker/2:008:31:152
31237992211,2cyclictest31650-21kworker/2:207:33:472
3123099222,14cyclictest86850irq/129-enp1s0-12:01:091
31230992218,3cyclictest0-21swapper/112:22:231
31230992214,3cyclictest0-21swapper/108:55:181
31230992214,3cyclictest0-21swapper/107:22:161
31230992213,6cyclictest14473-21awk12:35:201
3123099221,16cyclictest0-21swapper/110:32:401
3123099221,16cyclictest0-21swapper/108:16:081
3123099221,16cyclictest0-21swapper/108:16:081
3123099221,15cyclictest0-21swapper/110:20:021
3123099221,13cyclictest0-21swapper/110:43:401
3123099221,11cyclictest18456-21ntp_states07:50:211
3123099220,16cyclictest0-21swapper/108:45:371
3122699228,3cyclictest98372sendmail_mailst07:30:240
3122699226,6cyclictest0-21swapper/010:10:100
3124299216,7cyclictest780-21rngd10:00:053
3124299213,6cyclictest780-21rngd12:27:403
3124299213,5cyclictest0-21swapper/308:13:383
3124299213,3cyclictest780-21rngd09:21:433
3124299212,4cyclictest780-21rngd12:16:193
3124299211,5cyclictest0-21swapper/308:09:083
3124299211,3cyclictest0-21swapper/309:33:473
3123799217,4cyclictest12050irq/123-ahci[0012:25:002
3123799217,3cyclictest32-21ksoftirqd/207:55:472
3123799217,3cyclictest21780-21kworker/2:011:12:072
3123799213,5cyclictest32-21ksoftirqd/208:02:512
3123799211,2cyclictest31650-21kworker/2:207:28:492
31237992111,2cyclictest31650-21kworker/2:207:42:432
31237992110,3cyclictest31650-21kworker/2:207:47:552
3123099212,15cyclictest0-21swapper/108:42:071
3123099211,4cyclictest22594-21kworker/u8:210:55:141
31230992114,3cyclictest0-21swapper/112:31:441
31230992114,3cyclictest0-21swapper/112:31:441
31230992113,3cyclictest0-21swapper/110:10:091
3123099211,18cyclictest0-21swapper/111:09:031
3123099211,18cyclictest0-21swapper/110:00:161
3123099211,15cyclictest0-21swapper/111:31:151
3123099211,15cyclictest0-21swapper/110:35:501
3123099211,15cyclictest0-21swapper/109:53:091
3123099211,15cyclictest0-21swapper/108:36:381
3123099211,15cyclictest0-21swapper/107:45:361
3123099211,15cyclictest0-21swapper/107:39:261
3123099210,5cyclictest0-21swapper/109:56:131
3123099210,16cyclictest0-21swapper/111:27:041
3123099210,16cyclictest0-21swapper/111:27:041
3122699219,3cyclictest20004-21diskmemload10:15:240
3122699217,4cyclictest0-21swapper/011:00:500
3122699211,7cyclictest0-21swapper/009:30:200
3122699211,4cyclictest0-21swapper/010:04:570
187872211,3sleep118843-21kernelversion09:05:171
3124299209,3cyclictest28393-21/usr/sbin/munin08:15:293
3124299209,3cyclictest28393-21/usr/sbin/munin08:15:293
3124299206,7cyclictest780-21rngd10:30:373
3124299205,7cyclictest6374-21/usr/sbin/munin07:25:153
3124299205,7cyclictest6247-21cpuspeed_turbos11:05:103
3124299202,6cyclictest780-21rngd09:38:013
3124299201,6cyclictest0-21swapper/310:35:233
3123799207,5cyclictest21780-21kworker/2:009:43:512
3123799205,8cyclictest21780-21kworker/2:010:19:392
3123799203,9cyclictest32-21ksoftirqd/207:13:402
3123799201,11cyclictest21780-21kworker/2:009:15:172
3123799201,11cyclictest21780-21kworker/2:009:15:172
3123099202,17cyclictest0-21swapper/107:14:321
3123099201,9cyclictest0-21swapper/111:52:481
31230992016,3cyclictest0-21swapper/107:34:371
31230992016,2cyclictest0-21swapper/109:30:171
31230992013,2cyclictest0-21swapper/108:05:261
31230992012,3cyclictest0-21swapper/111:35:551
31230992012,3cyclictest0-21swapper/111:13:231
3123099201,17cyclictest0-21swapper/108:00:521
3123099201,15cyclictest0-21swapper/111:41:421
3123099201,15cyclictest0-21swapper/107:58:101
3123099200,15cyclictest0-21swapper/107:28:301
3122699209,3cyclictest19418-21/usr/sbin/munin11:35:120
3122699207,4cyclictest0-21swapper/012:05:110
3122699206,6cyclictest0-21swapper/010:47:470
3122699206,4cyclictest0-21swapper/009:14:350
3122699205,12cyclictest17855-21grep11:30:200
3122699203,6cyclictest0-21swapper/011:44:040
3122699203,14cyclictest5328-21sed12:15:170
31226992013,5cyclictest0-21swapper/009:15:210
31226992013,5cyclictest0-21swapper/009:15:210
3122699201,10cyclictest0-21swapper/011:58:240
3122699200,2cyclictest86550irq/126-enp1s0-11:15:240
3124299191,6cyclictest31034-21/usr/sbin/munin12:00:153
3124299191,11cyclictest2730-21sshd09:40:413
3123799198,2cyclictest21780-21kworker/2:008:19:432
3123799198,2cyclictest21780-21kworker/2:008:19:432
3123799197,2cyclictest21780-21kworker/2:010:05:592
3123799196,7cyclictest21780-21kworker/2:010:25:232
3123799195,3cyclictest21780-21kworker/2:008:05:052
3123799194,3cyclictest30-21rcuc/209:03:472
3123799194,2cyclictest21780-21kworker/2:010:50:192
31237991914,3cyclictest3909-21grep09:45:182
31237991913,3cyclictest22261-21tail11:40:152
3122699198,3cyclictest20004-21diskmemload11:48:310
3122699197,4cyclictest0-21swapper/009:20:330
3122699195,5cyclictest0-21swapper/010:35:520
3122699194,3cyclictest0-21swapper/012:14:090
3122699192,12cyclictest86550irq/126-enp1s0-10:51:420
3122699191,3cyclictest0-21swapper/011:52:360
3122699191,10cyclictest18706-21/usr/sbin/munin10:20:230
3122699191,10cyclictest14634-21snmpd08:53:130
3122699191,10cyclictest14634-21snmpd07:53:010
3122699191,10cyclictest0-21swapper/010:32:190
3122699191,10cyclictest0-21swapper/010:30:010
3122699191,10cyclictest0-21swapper/009:54:410
3122699190,9cyclictest0-21swapper/012:27:190
141962190,2sleep20-21swapper/208:55:162
3124299185,5cyclictest780-21rngd07:12:533
3124299184,5cyclictest0-21swapper/309:12:153
3124299183,7cyclictest780-21rngd08:02:303
3124299183,5cyclictest780-21rngd08:57:413
31242991812,3cyclictest18314-21cat07:50:193
3124299181,14cyclictest12991-21/usr/sbin/munin07:40:133
31237991811,2cyclictest86750irq/128-enp1s0-12:25:412
3122699185,11cyclictest11-21rcuc/012:36:420
3122699183,8cyclictest86550irq/126-enp1s0-10:41:200
3122699182,13cyclictest19719-21grep07:55:110
3122699181,9cyclictest0-21swapper/009:56:190
3122699181,9cyclictest0-21swapper/009:47:160
3122699181,4cyclictest6374-21/usr/sbin/munin07:25:210
3122699181,4cyclictest14634-21snmpd08:38:100
31226991813,3cyclictest11839-21ntpq07:35:210
3122699181,10cyclictest14634-21snmpd09:28:200
300822181,2sleep00-21swapper/008:16:380
300822181,2sleep00-21swapper/008:16:380
175942181,2sleep00-21swapper/009:02:350
146162181,2sleep00-21swapper/007:40:460
3124299171,8cyclictest14462-21cpuspeed_turbos10:10:143
3123799175,3cyclictest86550irq/126-enp1s0-11:20:132
3122699176,2cyclictest25547-21grep08:05:250
3122699171,6cyclictest0-21swapper/011:20:560
3122699171,6cyclictest0-21swapper/008:00:140
31226991712,3cyclictest5257-21open_inodes07:20:210
3122699171,10cyclictest0-21swapper/008:40:180
3124299162,5cyclictest0-21swapper/309:04:393
3124299161,13cyclictest18019-21/usr/sbin/munin09:05:163
3122699161,6cyclictest15189-21/usr/sbin/munin07:45:220
31226991612,3cyclictest0-21swapper/008:30:040
31226991610,3cyclictest18320-21expr09:05:110
3122699160,10cyclictest0-21swapper/008:25:090
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional