You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-15 - 23:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack1slot6s.osadl.org (updated Mon Sep 15, 2025 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
199189910641,20cyclictest23417-21kworker/u8:109:30:541
199169910437,12cyclictest0-21swapper/010:10:360
199169910142,17cyclictest24515-21awk10:05:220
267042692,57sleep00-21swapper/007:51:560
19918996942,12cyclictest0-21swapper/110:00:351
19918996641,9cyclictest23796-21tail06:40:441
1991899663,46cyclictest0-21swapper/107:00:441
19918996541,11cyclictest0-21swapper/110:50:341
19916996537,12cyclictest0-21swapper/010:05:360
19918996440,12cyclictest0-21swapper/105:51:061
19916996435,13cyclictest0-21swapper/009:00:250
19918996348,10cyclictest0-21swapper/110:25:231
19918996339,12cyclictest0-21swapper/105:26:081
106772633,52sleep00-21swapper/009:35:380
19918996239,11cyclictest0-21swapper/110:35:481
19918996239,11cyclictest0-21swapper/109:55:381
19918996238,12cyclictest0-21swapper/107:25:291
19918996238,12cyclictest0-21swapper/105:45:571
19918996138,11cyclictest0-21swapper/106:55:551
19918996138,11cyclictest0-21swapper/106:55:551
19916996138,11cyclictest0-21swapper/006:55:530
19916996138,11cyclictest0-21swapper/006:55:530
19916996136,12cyclictest0-21swapper/008:30:200
223099600,4rtkit-daemon2229-21rtkit-daemon05:25:503
19918996037,11cyclictest0-21swapper/106:45:441
19916996037,11cyclictest0-21swapper/010:35:400
19916996036,12cyclictest0-21swapper/005:30:470
19916996036,12cyclictest0-21swapper/005:30:470
19916996033,12cyclictest0-21swapper/007:43:470
19916996032,14cyclictest0-21swapper/007:05:490
19916995936,10cyclictest3228-21runrttasks09:20:250
1991699593,50cyclictest0-21swapper/005:40:480
19920995827,25cyclictest28899-21df_abs09:05:383
1991899582,51cyclictest0-21swapper/106:25:321
1991899582,49cyclictest0-21swapper/110:40:531
19916995836,10cyclictest0-21swapper/006:00:560
19916995834,12cyclictest0-21swapper/010:50:440
19916995834,11cyclictest0-21swapper/010:33:360
19916995834,11cyclictest0-21swapper/006:25:570
1991699583,49cyclictest0-21swapper/006:31:150
19920995736,8cyclictest28315-21iostat_ios05:40:583
19916995735,10cyclictest0-21swapper/008:50:460
19916995733,11cyclictest0-21swapper/010:20:330
19916995732,13cyclictest0-21swapper/009:10:270
1991699573,49cyclictest0-21swapper/009:25:480
1991699572,48cyclictest0-21swapper/009:35:240
47032563,45sleep00-21swapper/009:20:550
223099560,5rtkit-daemon2229-21rtkit-daemon09:20:553
223099560,5rtkit-daemon2229-21rtkit-daemon07:23:073
223099560,5rtkit-daemon2229-21rtkit-daemon06:23:102
1991699564,47cyclictest0-21swapper/007:45:490
19916995634,10cyclictest0-21swapper/007:25:290
19916995633,11cyclictest0-21swapper/005:40:020
19916995632,12cyclictest0-21swapper/007:00:400
1991699563,47cyclictest0-21swapper/010:25:370
1991699563,47cyclictest0-21swapper/010:25:370
1991699563,47cyclictest0-21swapper/008:21:270
74512553,12sleep20-21swapper/209:26:012
269532553,12sleep20-21swapper/209:00:502
223099550,5rtkit-daemon2229-21rtkit-daemon09:05:482
223099550,5rtkit-daemon2229-21rtkit-daemon07:45:592
223099540,5rtkit-daemon2229-21rtkit-daemon09:40:472
223099540,5rtkit-daemon2229-21rtkit-daemon09:40:472
223099540,5rtkit-daemon2229-21rtkit-daemon08:10:432
223099540,5rtkit-daemon2229-21rtkit-daemon06:15:552
19919995436,11cyclictest22134-21sed06:35:512
19919995435,12cyclictest8043-21tune2fs10:40:422
1991899545,27cyclictest0-21swapper/105:40:571
1991899544,26cyclictest0-21swapper/108:50:371
1991899543,44cyclictest9329-21munin-node06:11:021
1991899543,44cyclictest9329-21munin-node06:11:021
1991899543,43cyclictest25980-21munin-node09:00:491
19916995431,11cyclictest0-21swapper/007:15:430
133892543,43sleep00-21swapper/009:40:420
133892543,43sleep00-21swapper/009:40:420
81562533,43sleep00-21swapper/006:06:000
223099530,5rtkit-daemon2229-21rtkit-daemon07:35:293
223099530,5rtkit-daemon2229-21rtkit-daemon06:55:532
223099530,5rtkit-daemon2229-21rtkit-daemon06:55:532
223099530,5rtkit-daemon2229-21rtkit-daemon05:46:043
223099530,4rtkit-daemon2229-21rtkit-daemon10:24:232
19920995333,9cyclictest0-21swapper/307:10:583
19919995338,9cyclictest18147-21grep08:41:012
1991899535,26cyclictest0-21swapper/109:10:391
19918995338,10cyclictest0-21swapper/108:25:261
19918995334,11cyclictest0-21swapper/109:45:391
19918995310,22cyclictest4328-21awk06:00:321
1991699533,44cyclictest1160-21snmpd09:45:320
160702534,42sleep00-21swapper/006:22:570
159462533,43sleep10-21swapper/108:37:151
112962533,44sleep10-21swapper/108:28:531
235712523,42sleep10-21swapper/107:45:531
223099520,5rtkit-daemon2229-21rtkit-daemon09:50:473
223099520,5rtkit-daemon2229-21rtkit-daemon09:45:243
223099520,5rtkit-daemon2229-21rtkit-daemon09:45:243
223099520,5rtkit-daemon2229-21rtkit-daemon09:15:262
223099520,5rtkit-daemon2229-21rtkit-daemon08:25:502
223099520,5rtkit-daemon2229-21rtkit-daemon07:10:302
223099520,5rtkit-daemon2229-21rtkit-daemon06:50:542
223099520,5rtkit-daemon2229-21rtkit-daemon06:40:472
223099520,4rtkit-daemon2229-21rtkit-daemon06:25:472
19919995235,11cyclictest5393-21gltestperf06:00:492
19919995235,11cyclictest21225-21cat05:25:582
1991899523,44cyclictest0-21swapper/108:00:581
1991699522,42cyclictest16193-21latency_hist08:40:260
172112522,9sleep30-21swapper/309:48:473
26642513,40sleep00-21swapper/008:10:460
223099510,5rtkit-daemon2229-21rtkit-daemon09:19:033
223099510,5rtkit-daemon2229-21rtkit-daemon07:01:162
223099510,4rtkit-daemon2229-21rtkit-daemon08:35:422
19920995136,9cyclictest29601-21munin-node08:00:543
19920995131,9cyclictest0-21swapper/309:02:383
19920995127,10cyclictest18640-21if_eth007:35:513
19920995127,10cyclictest18640-21if_eth007:35:513
1991999514,38cyclictest0-21swapper/207:35:522
1991999514,38cyclictest0-21swapper/207:35:522
19919995135,9cyclictest7817-21grep06:05:532
19919995135,11cyclictest25590-21grep10:05:422
19919995133,9cyclictest28097-21grep05:40:552
19919995132,12cyclictest9850-21unixbench_singl07:16:032
19919995132,11cyclictest30877-21needreboot05:45:592
1991899514,41cyclictest19789-21runrttasks07:40:091
1991899514,41cyclictest19789-21runrttasks07:40:091
19918995135,11cyclictest0-21swapper/107:11:001
19918995134,11cyclictest0-21swapper/108:05:451
1991899513,42cyclictest0-21swapper/110:25:511
1991899513,42cyclictest0-21swapper/110:25:511
1991699512,43cyclictest0-21swapper/008:50:260
223099500,5rtkit-daemon2229-21rtkit-daemon10:30:232
223099500,5rtkit-daemon2229-21rtkit-daemon10:30:232
223099500,5rtkit-daemon2229-21rtkit-daemon10:10:462
223099500,5rtkit-daemon2229-21rtkit-daemon07:45:292
223099500,5rtkit-daemon2229-21rtkit-daemon05:55:482
223099500,4rtkit-daemon2229-21rtkit-daemon10:45:522
223099500,4rtkit-daemon2229-21rtkit-daemon10:40:222
223099500,4rtkit-daemon2229-21rtkit-daemon08:45:492
223099500,4rtkit-daemon2229-21rtkit-daemon07:10:532
223099500,4rtkit-daemon2229-21rtkit-daemon06:15:332
223099500,4rtkit-daemon2229-21rtkit-daemon06:15:332
1992099505,27cyclictest0-21swapper/308:25:453
19920995030,9cyclictest0-21swapper/309:33:233
19920995029,8cyclictest6668-21iostat_ios09:25:483
19919995036,9cyclictest11298-21gltestperf09:35:442
19919995034,10cyclictest7501-21tune2fs08:20:482
19919995033,10cyclictest25509-21fschecks_count07:50:472
19919995033,10cyclictest16353-21cat09:45:492
1991899505,39cyclictest0-21swapper/107:30:581
1991899504,40cyclictest0-21swapper/110:30:401
1991899504,40cyclictest0-21swapper/109:35:531
1991899504,38cyclictest0-21swapper/106:25:431
1991899504,27cyclictest0-21swapper/109:25:291
1991899504,26cyclictest25037-21python08:55:551
19918995032,12cyclictest0-21swapper/110:05:451
1991899503,20cyclictest0-21swapper/106:50:471
1991899502,42cyclictest0-21swapper/107:50:561
1991699503,41cyclictest0-21swapper/009:55:390
1991699503,41cyclictest0-21swapper/007:30:410
1991699503,41cyclictest0-21swapper/006:11:050
1991699503,41cyclictest0-21swapper/006:11:050
143242503,40sleep20-21swapper/210:55:212
265742492,40sleep30-21swapper/310:06:213
223099490,5rtkit-daemon2229-21rtkit-daemon10:30:452
223099490,5rtkit-daemon2229-21rtkit-daemon06:30:542
220182493,40sleep10-21swapper/107:43:021
19919994932,10cyclictest1365-21unixbench_multi05:51:072
1991999492,42cyclictest0-21swapper/206:50:012
1991899494,38cyclictest0-21swapper/109:50:341
19918994934,10cyclictest0-21swapper/105:35:451
19918994933,10cyclictest0-21swapper/110:10:531
19918994931,11cyclictest0-21swapper/106:35:431
1991899493,41cyclictest0-21swapper/109:06:081
1991899493,41cyclictest0-21swapper/106:30:461
1991899493,39cyclictest0-21swapper/108:10:561
1991699493,41cyclictest7464-21telnet10:40:350
1991699493,40cyclictest0-21swapper/007:10:420
1991699493,39cyclictest0-21swapper/009:00:450
1991699493,39cyclictest0-21swapper/006:15:480
19916994925,11cyclictest0-21swapper/006:46:090
1991699492,42cyclictest31753-21latency_hist08:05:270
1991699492,41cyclictest0-21swapper/008:15:410
1991699492,41cyclictest0-21swapper/008:15:410
185322492,9sleep30-21swapper/306:28:563
149062493,9sleep10-21swapper/107:28:441
242012482,38sleep30-21swapper/310:01:333
223099480,4rtkit-daemon2229-21rtkit-daemon08:31:272
19920994827,8cyclictest1122-21lldpd08:58:383
19920994825,9cyclictest14212-21telnet06:20:423
19920994824,8cyclictest361ktimersoftd/308:40:573
19920994821,10cyclictest361ktimersoftd/310:45:533
19920994818,14cyclictest0-21swapper/307:40:413
1991999488,29cyclictest0-21swapper/209:15:382
1991999484,38cyclictest11700-21tail07:20:562
19919994831,11cyclictest22863-21grep05:30:472
19919994831,11cyclictest22863-21grep05:30:472
19919994830,12cyclictest4059-21gltestperf09:20:432
19919994822,15cyclictest0-21swapper/209:30:522
19918994835,7cyclictest29369-21munin-run08:00:271
19918994832,11cyclictest0-21swapper/107:10:071
1991899483,40cyclictest24683-21awk05:35:331
1991899483,40cyclictest24683-21awk05:35:331
1991699483,40cyclictest0-21swapper/008:30:420
1991699483,40cyclictest0-21swapper/006:41:320
1991699483,40cyclictest0-21swapper/006:36:010
1991699482,9cyclictest19908-21cyclictest06:51:050
1991699482,41cyclictest0-21swapper/009:50:480
1991699482,40cyclictest19908-21cyclictest10:15:370
196202486,9sleep00-21swapper/007:37:580
196202486,9sleep00-21swapper/007:37:580
147922483,40sleep10-21swapper/109:43:211
147922483,40sleep10-21swapper/109:43:211
46502472,9sleep30-21swapper/310:33:463
317932472,9sleep20-21swapper/208:05:282
267642472,39sleep20-21swapper/205:36:482
223099470,4rtkit-daemon2229-21rtkit-daemon07:16:083
1992099474,35cyclictest7984-21diskstats10:40:413
19920994727,7cyclictest4319-21munin-run06:00:323
19920994726,8cyclictest28281-21missed_timers10:10:483
19920994726,8cyclictest1160-21snmpd08:20:193
19920994726,8cyclictest1160-21snmpd08:20:193
19920994723,9cyclictest28445-21munin-node06:50:573
1991999474,35cyclictest0-21swapper/207:55:562
1991999474,34cyclictest0-21swapper/207:25:542
1991999474,33cyclictest0-21swapper/210:16:222
1991899474,24cyclictest11772-21aten2.4-expect06:15:441
19918994732,10cyclictest0-21swapper/109:15:571
19918994731,11cyclictest0-21swapper/108:15:561
19918994731,11cyclictest0-21swapper/108:15:561
19918994731,11cyclictest0-21swapper/106:09:581
1991899473,12cyclictest0-21swapper/108:30:381
1991699473,40cyclictest0-21swapper/010:45:340
1991699473,39cyclictest0-21swapper/008:40:400
1991699473,39cyclictest0-21swapper/005:54:530
19916994729,11cyclictest19917-21kworker/u8:105:25:330
231202462,9sleep30-21swapper/306:36:263
22182462,9sleep30-21swapper/310:28:383
22182462,9sleep30-21swapper/310:28:383
218292462,38sleep30-21swapper/309:56:413
205092463,37sleep020585-21df_abs05:25:500
19920994626,7cyclictest1160-21snmpd08:10:173
19920994624,9cyclictest5016-21munin-node10:35:503
19920994624,8cyclictest18815-21munin-node08:45:463
1991999465,35cyclictest0-21swapper/207:30:452
1991999465,32cyclictest0-21swapper/208:50:372
1991999464,33cyclictest0-21swapper/209:50:412
1991999464,32cyclictest0-21swapper/209:55:542
19919994630,9cyclictest23593-21munin-node08:55:462
19919994617,20cyclictest23975-21timerwakeupswit10:00:552
1991899469,32cyclictest5422-21awk09:25:241
1991899466,14cyclictest0-21swapper/106:03:421
19918994630,12cyclictest0-21swapper/110:45:421
19918994630,11cyclictest0-21swapper/108:45:561
1991899463,36cyclictest0-21swapper/107:15:511
1991899462,38cyclictest0-21swapper/108:45:261
1991699463,39cyclictest0-21swapper/005:46:080
1991699463,26cyclictest0-21swapper/008:05:490
19916994629,12cyclictest0-21swapper/007:56:560
1991699462,39cyclictest0-21swapper/009:05:420
125062463,9sleep30-21swapper/309:39:413
42592452,9sleep00-21swapper/005:59:550
268462452,9sleep30-21swapper/307:53:493
232932452,9sleep30-21swapper/308:54:423
19920994529,10cyclictest7415-21df_abs08:20:463
19920994523,8cyclictest12228-21df_inode08:30:433
19920994522,9cyclictest26060-21munin-node06:45:493
19920994522,9cyclictest1160-21snmpd10:50:433
19920994522,9cyclictest1160-21snmpd06:04:573
19920994519,11cyclictest0-21swapper/306:20:263
19919994520,15cyclictest0-21swapper/208:00:402
1991899453,38cyclictest0-21swapper/110:18:581
198212454,6sleep10-21swapper/105:24:461
149772452,37sleep00-21swapper/007:29:390
90692442,35sleep30-21swapper/306:10:183
305712442,9sleep30-21swapper/309:10:263
1992099446,32cyclictest0-21swapper/307:30:293
19920994427,10cyclictest25657-21fschecks_time05:35:523
19920994421,7cyclictest4808-21switchtime07:06:033
19920994419,11cyclictest30134-21munin-node10:20:443
19919994430,9cyclictest5142-21grep08:15:472
19919994430,9cyclictest5142-21grep08:15:472
1992099436,32cyclictest0-21swapper/305:25:343
1992099434,11cyclictest0-21swapper/310:20:053
1992099433,33cyclictest0-21swapper/308:10:393
19920994321,8cyclictest14613-21df_inode08:35:403
19920994319,11cyclictest0-21swapper/306:41:003
19920994315,8cyclictest14279-21kworker/3:207:45:393
3762422,9sleep30-21swapper/306:58:513
3762422,9sleep30-21swapper/306:58:513
1992099426,29cyclictest0-21swapper/307:01:003
19920994229,8cyclictest9619-21cpuspeed_turbos06:10:483
19920994229,8cyclictest9619-21cpuspeed_turbos06:10:483
19920994227,9cyclictest20245-21proc_pri06:31:023
19920994213,13cyclictest0-21swapper/308:00:283
19920994212,11cyclictest0-21swapper/305:30:453
19920994212,11cyclictest0-21swapper/305:30:453
1991999424,30cyclictest0-21swapper/205:25:342
15832422,9sleep30-21swapper/305:52:533
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional