You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-14 - 14:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sun Dec 14, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
48542240164,24sleep20-21swapper/207:07:202
48542240164,24sleep20-21swapper/207:07:192
47982232162,57sleep10-21swapper/107:06:351
47982232162,57sleep10-21swapper/107:06:341
48402203167,24sleep30-21swapper/307:07:083
48402203167,24sleep30-21swapper/307:07:083
48622201164,24sleep00-21swapper/007:07:260
48622201164,24sleep00-21swapper/007:07:260
875521710,3sleep2515599cyclictest10:55:192
82921660,5sleep1515099cyclictest09:25:111
82921660,5sleep1515099cyclictest09:25:111
1144221320,5sleep3516299cyclictest07:20:213
5150997269,2cyclictest0-21swapper/109:11:091
5942560,5sleep3595-21needreboot11:50:183
514499379,13cyclictest14891-21idleruntime-cro12:25:020
514499345,13cyclictest26830-21munin-run10:25:010
515599331,13cyclictest0-21swapper/212:30:112
515599320,14cyclictest0-21swapper/208:16:382
515599320,14cyclictest0-21swapper/208:16:372
516299305,5cyclictest87450irq/129-enp1s0-08:45:273
516299301,11cyclictest0-21swapper/310:59:113
515599301,13cyclictest0-21swapper/210:49:462
515599301,12cyclictest31447-21/usr/sbin/munin10:35:142
515599301,12cyclictest31447-21/usr/sbin/munin10:35:132
514499301,13cyclictest1680-21munin-run11:55:010
515599291,10cyclictest0-21swapper/211:12:482
515599291,10cyclictest0-21swapper/209:48:312
515599291,10cyclictest0-21swapper/209:27:272
515599291,10cyclictest0-21swapper/209:27:272
515599290,12cyclictest0-21swapper/211:53:592
514499298,10cyclictest0-21swapper/008:45:010
514499296,11cyclictest21821-21basename09:00:010
516299281,15cyclictest774-21dbus-daemon12:10:323
516299281,10cyclictest0-21swapper/307:52:313
515599281,12cyclictest0-21swapper/208:25:442
515599281,10cyclictest0-21swapper/207:16:152
516299272,14cyclictest20857-21munin-run11:25:013
516299272,12cyclictest0-21swapper/312:22:593
516299270,13cyclictest0-21swapper/308:05:163
515599271,9cyclictest0-21swapper/212:05:412
515599271,12cyclictest0-21swapper/209:42:032
515599271,10cyclictest0-21swapper/212:19:172
516299260,17cyclictest0-21swapper/307:30:263
516299260,10cyclictest0-21swapper/310:45:293
515599261,11cyclictest0-21swapper/211:36:332
515599261,11cyclictest0-21swapper/210:22:192
515599261,11cyclictest0-21swapper/207:50:092
515599261,10cyclictest0-21swapper/209:39:472
515599261,10cyclictest0-21swapper/207:22:212
514499261,13cyclictest18666-21awk11:20:010
164522260,2sleep2311ktimersoftd/212:25:232
516299255,6cyclictest4025-1kworker/3:0H08:20:123
516299254,12cyclictest26439-21/usr/sbin/munin09:10:153
5162992510,2cyclictest0-21swapper/309:55:113
516299250,4cyclictest12050irq/123-ahci[0010:30:013
515599251,9cyclictest0-21swapper/211:42:032
515599251,9cyclictest0-21swapper/207:36:252
515599251,8cyclictest0-21swapper/210:50:542
515599251,11cyclictest26208-21diskmemload10:15:212
515599251,11cyclictest0-21swapper/211:49:212
515599251,11cyclictest0-21swapper/211:49:212
515599251,11cyclictest0-21swapper/210:32:272
515599251,11cyclictest0-21swapper/210:32:272
515599251,11cyclictest0-21swapper/207:14:182
515599251,10cyclictest0-21swapper/211:32:212
515599251,10cyclictest0-21swapper/211:00:102
515599251,10cyclictest0-21swapper/209:53:592
515599251,10cyclictest0-21swapper/209:22:532
515599251,10cyclictest0-21swapper/208:58:322
515599251,10cyclictest0-21swapper/207:56:372
515599251,10cyclictest0-21swapper/207:41:592
515599250,11cyclictest0-21swapper/211:55:032
515599250,11cyclictest0-21swapper/210:06:452
515599250,11cyclictest0-21swapper/207:46:272
5150992510,13cyclictest32680-21kworker/u8:111:13:071
514499259,13cyclictest24204-21unixbench_singl10:15:260
514499251,12cyclictest19002-21munin-run07:40:020
516299242,15cyclictest11081-1kworker/3:0H12:30:073
5162992411,3cyclictest30179-21expr11:45:133
5162992411,3cyclictest30179-21expr11:45:133
516299241,12cyclictest800-21snmpd10:40:133
516299240,16cyclictest0-21swapper/309:25:123
516299240,16cyclictest0-21swapper/309:25:123
516299240,10cyclictest0-21swapper/311:40:153
515599241,15cyclictest0-21swapper/209:06:382
515599241,12cyclictest0-21swapper/208:01:552
515599241,11cyclictest0-21swapper/212:03:172
515599241,11cyclictest0-21swapper/209:57:072
515599240,11cyclictest0-21swapper/207:30:212
515599240,10cyclictest32680-21kworker/u8:110:25:032
5150992418,3cyclictest19225-21diskstats11:20:141
514499241,10cyclictest22067-21strings11:25:180
516299234,5cyclictest87350irq/128-enp1s0-10:00:123
516299232,10cyclictest0-21swapper/310:30:223
516299232,10cyclictest0-21swapper/310:30:223
516299231,8cyclictest0-21swapper/312:08:323
516299231,14cyclictest0-21swapper/311:00:113
516299230,9cyclictest3991-21latency_hist12:00:023
516299230,14cyclictest0-21swapper/311:30:113
516299230,13cyclictest14599-21latency_hist07:30:013
515599237,3cyclictest0-21swapper/210:44:042
515599231,9cyclictest0-21swapper/209:33:532
515599231,8cyclictest26208-21diskmemload10:04:372
515599231,12cyclictest0-21swapper/208:46:482
515599231,10cyclictest0-21swapper/208:13:472
515599230,9cyclictest0-21swapper/212:10:152
515599230,9cyclictest0-21swapper/210:12:192
515599230,8cyclictest0-21swapper/211:07:182
515599230,10cyclictest0-21swapper/211:16:282
515099232,14cyclictest87350irq/128-enp1s0-11:19:311
5150992311,3cyclictest0-21swapper/108:30:131
514499231,9cyclictest0-21swapper/008:50:210
514499230,21cyclictest0-21swapper/009:08:300
516299226,5cyclictest87150irq/126-enp1s0-11:35:143
516299225,3cyclictest0-21swapper/311:13:473
516299223,5cyclictest12050irq/123-ahci[0012:38:373
516299223,10cyclictest87150irq/126-enp1s0-09:40:143
516299222,10cyclictest0-21swapper/309:30:163
516299221,12cyclictest8629-21munin-run08:30:013
516299220,9cyclictest9468-21/usr/sbin/munin09:45:193
516299220,15cyclictest0-21swapper/312:30:013
516299220,15cyclictest0-21swapper/310:37:443
516299220,15cyclictest0-21swapper/310:37:443
516299220,14cyclictest18133-21timerwakeupswit11:15:243
516299220,12cyclictest87350irq/128-enp1s0-07:35:163
516299220,12cyclictest28638-21/usr/sbin/munin09:15:243
515599221,9cyclictest6628-21cat08:25:022
515599221,9cyclictest0-21swapper/211:27:042
515599221,9cyclictest0-21swapper/208:36:182
515599221,14cyclictest0-21swapper/207:25:332
515599221,11cyclictest0-21swapper/208:50:182
515599220,9cyclictest0-21swapper/212:20:072
515099228,12cyclictest19245-21kworker/u8:010:56:511
515099227,13cyclictest32680-21kworker/u8:107:50:011
5150992219,2cyclictest0-21swapper/108:20:161
515099221,18cyclictest21993-21memory11:25:181
515099221,15cyclictest0-21swapper/107:10:051
515099220,10cyclictest0-21swapper/108:00:191
514499228,3cyclictest22672/usr/sbin/munin08:15:100
514499228,3cyclictest22672/usr/sbin/munin08:15:100
514499227,12cyclictest7267-21/usr/sbin/munin09:40:130
514499221,9cyclictest0-21swapper/010:45:250
514499221,10cyclictest8941-21ntpq10:55:200
514499221,10cyclictest28429-21ethtool11:40:170
514499221,10cyclictest13514-21gzip11:05:220
516299212,7cyclictest0-21swapper/312:04:393
5162992111,7cyclictest31616-21if_err_enp2s009:20:183
5162992111,7cyclictest23140-21ls09:00:213
516299211,12cyclictest800-21snmpd10:10:063
516299210,8cyclictest0-21swapper/311:05:173
516299210,4cyclictest0-21swapper/312:15:183
516299210,15cyclictest0-21swapper/310:07:303
516299210,14cyclictest11081-1kworker/3:0H10:21:373
516299210,14cyclictest0-21swapper/308:10:183
515599216,3cyclictest0-21swapper/209:19:292
515599216,3cyclictest0-21swapper/209:00:062
515599216,3cyclictest0-21swapper/208:41:322
515099217,12cyclictest23-21ksoftirqd/111:04:021
515099211,9cyclictest1295-21sendmail_mailst08:10:231
5150992113,6cyclictest19245-1kworker/u8:007:30:061
515099211,19cyclictest0-21swapper/109:57:171
515099211,18cyclictest0-21swapper/109:30:171
515099211,18cyclictest0-21swapper/109:02:211
515099211,17cyclictest26846-21sleep10:25:011
5150992110,3cyclictest32680-21kworker/u8:110:49:031
514499217,13cyclictest0-21swapper/007:35:020
514499212,11cyclictest10434-21/usr/sbin/munin07:20:170
516299209,3cyclictest2251-21/usr/sbin/munin08:15:223
516299209,3cyclictest2251-21/usr/sbin/munin08:15:223
516299205,5cyclictest87150irq/126-enp1s0-09:07:393
516299202,5cyclictest22658-21/usr/sbin/munin10:15:263
516299201,7cyclictest9219-21sed07:15:193
516299201,4cyclictest87150irq/126-enp1s0-07:45:203
5162992011,6cyclictest19606-21latency_hist08:55:033
516299201,10cyclictest23043-21latency_hist11:30:023
516299200,9cyclictest0-21swapper/308:44:513
516299200,8cyclictest0-21swapper/307:55:393
516299200,6cyclictest0-21swapper/308:59:553
516299200,18cyclictest0-21swapper/308:00:123
516299200,13cyclictest0-21swapper/308:30:223
516299200,10cyclictest0-21swapper/308:35:173
515599205,2cyclictest0-21swapper/212:38:332
515599201,8cyclictest32180-21sh08:10:002
515599201,12cyclictest0-21swapper/211:20:142
515099204,3cyclictest0-21swapper/112:20:411
5150992013,4cyclictest20809-21users07:40:251
515099201,18cyclictest0-21swapper/110:17:001
515099201,18cyclictest0-21swapper/107:23:471
515099201,17cyclictest0-21swapper/112:39:291
515099201,17cyclictest0-21swapper/111:54:501
515099201,11cyclictest12002-21df_inode09:50:121
515099200,18cyclictest0-21swapper/111:58:461
514499207,2cyclictest2893-21sshd10:40:310
514499205,3cyclictest14485-21/usr/sbin/munin11:10:110
514499201,6cyclictest10084-21/usr/sbin/munin11:00:150
514499201,16cyclictest19218-21sshd12:34:320
514499200,9cyclictest0-21swapper/009:25:250
514499200,9cyclictest0-21swapper/009:25:250
514499200,10cyclictest26533-21idleruntime07:55:160
516299192,14cyclictest5705-21sed09:35:153
5162991911,5cyclictest5794-21cpuspeed_turbos10:50:123
516299190,8cyclictest0-21swapper/309:50:173
515099199,1cyclictest0-21swapper/110:05:121
515099191,6cyclictest0-21swapper/109:49:311
5150991916,2cyclictest0-21swapper/112:17:471
5150991916,2cyclictest0-21swapper/110:44:031
515099191,5cyclictest0-21swapper/109:09:551
5150991915,2cyclictest6310-21gltestperf10:50:161
515099191,3cyclictest0-21swapper/110:13:241
515099191,2cyclictest0-21swapper/112:34:021
515099191,17cyclictest0-21swapper/110:00:091
515099191,17cyclictest0-21swapper/109:43:081
515099191,16cyclictest0-21swapper/111:05:401
515099191,16cyclictest0-21swapper/108:50:331
515099191,16cyclictest0-21swapper/108:05:431
515099191,16cyclictest0-21swapper/107:35:381
5150991911,2cyclictest16430-21strings08:45:191
515099191,10cyclictest20184-21diskstats08:55:131
515099190,17cyclictest0-21swapper/112:00:161
514499195,3cyclictest5342-21/usr/sbin/munin07:10:070
514499193,3cyclictest87350irq/128-enp1s0-12:05:310
514499192,14cyclictest26933-21grep11:35:240
514499191,5cyclictest16791-21sshd12:26:320
5144991914,3cyclictest13630-21turbostat.cron09:55:020
514499191,15cyclictest6063-21sshd08:20:300
514499190,9cyclictest0-21swapper/011:20:120
67212180,2sleep00-21swapper/009:36:040
516299181,10cyclictest87150irq/126-enp1s0-07:44:143
516299180,6cyclictest87150irq/126-enp1s0-07:10:103
515599185,6cyclictest0-21swapper/209:10:192
515599180,11cyclictest0-21swapper/208:30:462
515099181,9cyclictest0-21swapper/109:35:251
515099181,5cyclictest31447-21/usr/sbin/munin10:35:181
515099181,5cyclictest31447-21/usr/sbin/munin10:35:181
5150991815,2cyclictest0-21swapper/112:10:151
5150991815,2cyclictest0-21swapper/107:28:241
5150991814,3cyclictest0-21swapper/109:22:221
5150991814,3cyclictest0-21swapper/109:16:341
5150991813,3cyclictest2587-21df08:15:121
5150991813,3cyclictest2587-21df08:15:121
515099181,16cyclictest0-21swapper/112:28:041
515099181,16cyclictest0-21swapper/111:43:211
515099181,16cyclictest0-21swapper/111:36:411
515099181,16cyclictest0-21swapper/110:27:291
515099181,16cyclictest0-21swapper/108:43:471
515099181,16cyclictest0-21swapper/108:36:331
515099181,16cyclictest0-21swapper/108:25:411
515099181,16cyclictest0-21swapper/107:55:501
515099181,16cyclictest0-21swapper/107:51:471
515099181,16cyclictest0-21swapper/107:15:371
5150991811,4cyclictest6918-21gltestperf12:05:161
515099180,17cyclictest0-21swapper/110:31:431
515099180,17cyclictest0-21swapper/110:31:431
515099180,16cyclictest0-21swapper/111:46:041
515099180,16cyclictest0-21swapper/111:46:041
514499187,8cyclictest27159-21cpuspeed_turbos10:25:130
514499184,3cyclictest87350irq/128-enp1s0-08:25:160
514499183,3cyclictest5686-21/usr/sbin/munin10:50:140
514499182,4cyclictest5176-21ntp_states12:00:210
514499181,6cyclictest12266-21kworker/0:207:28:310
514499181,5cyclictest29866-21/usr/sbin/munin11:45:110
514499181,5cyclictest29866-21/usr/sbin/munin11:45:110
514499181,5cyclictest16061-21/usr/sbin/munin10:00:120
514499181,5cyclictest0-21swapper/008:00:140
5144991811,5cyclictest20137-21/usr/sbin/munin07:40:190
5144991811,4cyclictest11777-21ntpq12:15:220
514499180,9cyclictest0-21swapper/012:35:130
514499180,9cyclictest0-21swapper/010:35:110
514499180,9cyclictest0-21swapper/010:35:110
514499180,8cyclictest0-21swapper/009:17:160
514499180,8cyclictest0-21swapper/007:19:290
514499180,7cyclictest0-21swapper/011:57:310
200632180,2sleep00-21swapper/010:07:580
5150991714,2cyclictest0-21swapper/111:30:231
514499173,9cyclictest32527-21taskset09:21:340
514499171,6cyclictest0-21swapper/009:56:390
5144991713,3cyclictest0-21swapper/008:10:200
5144991712,3cyclictest20524-21cpuspeed_turbos10:10:110
5144991711,3cyclictest9379-21/usr/sbin/munin08:30:140
514499171,13cyclictest23693-21cpuspeed_turbos07:50:120
514499171,13cyclictest22033-21/usr/sbin/munin09:00:140
514499171,13cyclictest21421-21/usr/sbin/munin07:45:140
514499171,13cyclictest16495-21ntpq08:45:200
5144991711,3cyclictest10120-21sshd12:10:330
514499169,5cyclictest0-21swapper/009:10:130
514499162,11cyclictest9532-21cpuspeed_turbos09:45:120
5144991611,3cyclictest25041-21seq11:32:320
5144991610,3cyclictest11713-21gltestperf08:35:150
514499159,4cyclictest31290-21open_files08:05:200
514499151,3cyclictest30298-21ntp_kernel_pll_10:30:200
514499151,3cyclictest30298-21ntp_kernel_pll_10:30:190
514499149,3cyclictest2958-21cpuspeed_turbos09:30:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional