You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 23:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Wed Dec 10, 2025 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
231102240162,24sleep10-21swapper/107:06:071
231642236165,23sleep30-21swapper/307:06:513
234032205167,25sleep00-21swapper/007:09:540
233062202178,16sleep20-21swapper/207:08:432
421921750,4sleep02350499cyclictest08:50:160
2949321460,3sleep22351699cyclictest08:35:132
23516991153,109cyclictest13069-21sendmail_mailst07:55:242
23516996359,3cyclictest0-21swapper/210:15:412
186432440,1sleep20-21swapper/210:35:232
23504994111,14cyclictest27642-21munin-run09:45:010
255422360,4sleep081ktimersoftd/010:50:260
23516993633,2cyclictest0-21swapper/211:06:022
23504993617,16cyclictest32681-21munin-run11:10:010
2351699331,22cyclictest0-21swapper/211:58:042
2350499335,13cyclictest22615-21perf08:20:020
36932320,2sleep10-21swapper/111:15:201
2352199323,20cyclictest0-21swapper/307:15:273
23516993228,3cyclictest0-21swapper/209:33:302
23516993211,6cyclictest473-21hddtemp_smartct09:55:182
2350999321,29cyclictest0-21swapper/108:10:141
2351699314,12cyclictest7319-21cstates11:25:142
23516993111,4cyclictest86550irq/126-enp1s0-09:35:152
2350499311,14cyclictest2807-21munin-run07:35:000
2352199302,15cyclictest401ktimersoftd/312:25:023
2350499301,14cyclictest7553-21munin-run12:40:020
2351699298,5cyclictest31568-21grep09:50:272
2351699298,5cyclictest31568-21grep09:50:272
2351699297,9cyclictest7063-21perl10:10:172
2351699296,13cyclictest674-21taskset12:22:452
2351699295,9cyclictest14546-21/usr/sbin/munin11:40:172
2351699294,14cyclictest15306-21/usr/sbin/munin10:30:232
2351699292,15cyclictest13096-21gltestperf09:10:162
23516992910,7cyclictest19398-21df_inode09:25:122
2350999291,18cyclictest26619-21users08:25:261
2351699289,7cyclictest10924-21users11:30:272
2351699285,12cyclictest86750irq/128-enp1s0-10:26:382
2351699282,15cyclictest26948-21/usr/sbin/munin12:10:222
2351699281,11cyclictest0-21swapper/207:52:122
23516992810,5cyclictest5907-21df12:35:152
2350999281,17cyclictest11972-21ntp_states10:20:221
2350499280,13cyclictest7213-21basename07:45:000
2350499280,13cyclictest7213-21basename07:45:000
129632280,6sleep011-21rcuc/009:10:150
2351699278,5cyclictest9295-21sshd07:48:542
2351699277,11cyclictest11715-21irqstats10:20:182
2351699275,6cyclictest12236-21diskmemload12:26:132
2351699275,10cyclictest17916-21perf11:50:012
23516992711,6cyclictest1334-21/usr/sbin/munin11:10:172
2350999274,3cyclictest0-21swapper/111:13:571
2350499272,12cyclictest1875-21munin-run10:00:010
2352199264,15cyclictest401ktimersoftd/310:52:173
2351699264,12cyclictest27535-21chrt09:43:532
2351699264,12cyclictest14021-21/usr/sbin/munin08:00:172
2351699261,8cyclictest7881-21kworker/2:207:41:562
2351699261,8cyclictest7881-21kworker/2:207:41:562
23509992614,3cyclictest5800-21latency_hist08:55:021
23509992614,2cyclictest0-21swapper/107:48:421
2350499261,12cyclictest16005-21munin-run08:05:010
203432260,4sleep339-21rcuc/310:40:183
2352199251,16cyclictest0-21swapper/309:19:133
2352199250,19cyclictest0-21swapper/309:20:203
23516992516,5cyclictest27563-21smart_sda10:55:262
23516992513,3cyclictest86850irq/129-enp1s0-08:19:442
2351699251,12cyclictest24779-21hddtemp_smartct10:50:182
2351699251,12cyclictest0-21swapper/209:03:042
2351699251,12cyclictest0-21swapper/208:50:242
2351699251,12cyclictest0-21swapper/208:05:482
2351699251,11cyclictest20041-21df_inode10:40:162
2351699251,11cyclictest0-21swapper/207:29:202
2351699251,10cyclictest25366-21df08:25:142
2351699251,10cyclictest0-21swapper/209:08:422
2351699251,10cyclictest0-21swapper/208:57:002
2350999251,16cyclictest15604-21uptime08:00:271
2350499256,16cyclictest7843-21perf09:00:010
2350499251,11cyclictest12252-21awk09:10:010
2352199243,11cyclictest86550irq/126-enp1s0-09:55:223
2352199241,14cyclictest0-21swapper/310:17:303
2352199241,12cyclictest0-21swapper/307:40:583
2352199241,12cyclictest0-21swapper/307:40:583
2352199240,16cyclictest0-21swapper/312:25:133
2351699247,6cyclictest5857-21memory11:20:202
2351699245,5cyclictest27866-21/usr/sbin/munin09:45:182
2351699243,7cyclictest2097-21/usr/sbin/munin10:00:222
2351699243,13cyclictest26442-21taskset12:06:412
23516992414,7cyclictest20548-21latency_hist08:15:022
2351699241,11cyclictest0-21swapper/207:34:282
2350999248,10cyclictest0-21swapper/109:41:421
2350999242,11cyclictest5350-21munin-run12:35:011
23509992415,3cyclictest0-21swapper/108:15:101
2350999241,3cyclictest6784-21chrt11:23:351
2350999241,17cyclictest0-21swapper/112:05:031
2350999241,17cyclictest0-21swapper/110:42:361
2350999241,17cyclictest0-21swapper/110:31:381
2350999241,17cyclictest0-21swapper/110:15:171
2350999241,17cyclictest0-21swapper/107:34:281
2350999241,13cyclictest0-21swapper/111:08:231
2350999241,10cyclictest0-21swapper/109:56:271
2350999240,5cyclictest26511-21kworker/u8:212:22:081
2350499245,16cyclictest32029-21dump-pmu-power09:55:010
2350499245,16cyclictest32029-21dump-pmu-power09:55:010
2350499241,14cyclictest27356-21kworker/0:309:47:160
2352199237,8cyclictest0-21swapper/310:28:003
2352199235,11cyclictest0-21swapper/311:07:133
2352199234,7cyclictest0-21swapper/310:06:533
2352199232,13cyclictest39-21rcuc/311:30:183
23521992314,2cyclictest88250irq/135-enp2s0-09:08:563
2352199231,11cyclictest0-21swapper/309:13:163
2352199231,10cyclictest0-21swapper/311:58:023
2352199230,15cyclictest0-21swapper/310:39:473
2351699238,3cyclictest0-21swapper/207:15:562
2351699235,9cyclictest4051-21hddtemp_smartct12:30:172
23516992315,5cyclictest2727-21smart_sda08:45:272
23516992315,5cyclictest2727-21smart_sda08:45:272
23516992313,6cyclictest22393-21tune2fs10:45:182
2350999233,12cyclictest14634-21snmpd07:40:331
2350999233,12cyclictest14634-21snmpd07:40:331
2350999231,5cyclictest0-21swapper/112:38:531
2350999231,17cyclictest0-21swapper/109:53:101
2350999231,17cyclictest0-21swapper/109:53:091
2350999231,16cyclictest0-21swapper/112:10:031
2350999231,16cyclictest0-21swapper/110:39:441
2350999231,16cyclictest0-21swapper/109:16:511
23509992311,3cyclictest11599-21munin-run07:55:011
2350999231,10cyclictest0-21swapper/110:59:361
301892221,3sleep00-21swapper/011:01:270
2352199227,6cyclictest0-21swapper/311:23:493
2352199227,3cyclictest88050irq/133-enp2s0-12:33:543
2352199227,3cyclictest0-21swapper/308:25:363
2352199226,5cyclictest780-21rngd07:57:543
2352199222,10cyclictest5001-21munin-run07:40:013
2352199221,16cyclictest41-21ksoftirqd/310:55:183
2352199221,14cyclictest0-21swapper/309:42:253
2352199221,13cyclictest2908-21irqstats10:00:183
2351699228,5cyclictest2745-21/usr/sbin/munin11:15:192
2351699225,4cyclictest86550irq/126-enp1s0-08:30:252
2351699224,3cyclictest86550irq/126-enp1s0-10:05:232
2351699224,15cyclictest28821-21cpu07:20:142
2351699221,9cyclictest0-21swapper/208:41:442
2351699221,8cyclictest0-21swapper/208:20:172
23516992213,6cyclictest19118-21needreboot11:50:222
23516992213,6cyclictest18403-21/usr/sbin/munin09:20:262
23516992211,8cyclictest22866-21df12:00:152
23516992211,8cyclictest22866-21df12:00:152
2351699221,11cyclictest0-21swapper/207:13:502
2350999223,16cyclictest13066-21grep11:35:261
2350999222,11cyclictest13451-21diskstats10:25:141
2350999221,9cyclictest18841-21perf09:25:011
23509992215,2cyclictest0-21swapper/111:52:031
23509992215,2cyclictest0-21swapper/108:08:301
23509992214,3cyclictest0-21swapper/110:53:171
2350999221,16cyclictest0-21swapper/110:04:291
2350999221,15cyclictest0-21swapper/111:03:241
2350999221,15cyclictest0-21swapper/109:30:121
2350999221,15cyclictest0-21swapper/108:20:471
2350999221,11cyclictest28212-21diskstats09:45:141
2350999220,16cyclictest0-21swapper/111:40:201
2350499227,3cyclictest0-21swapper/011:56:070
2350499224,10cyclictest0-21swapper/009:25:200
23504992215,4cyclictest2509-21crond11:15:010
23504992210,3cyclictest23487-21users10:45:280
4022211,2sleep10-21swapper/107:28:571
2352199219,5cyclictest401ktimersoftd/310:30:213
2352199218,8cyclictest0-21swapper/309:26:163
2352199216,3cyclictest0-21swapper/312:10:163
2352199215,3cyclictest0-21swapper/308:47:463
2352199215,3cyclictest0-21swapper/308:47:463
2352199214,6cyclictest0-21swapper/309:54:313
2352199214,6cyclictest0-21swapper/309:54:313
2352199213,9cyclictest780-21rngd09:01:523
2352199211,7cyclictest0-21swapper/307:27:463
2352199211,4cyclictest0-21swapper/311:50:213
2352199211,3cyclictest0-21swapper/311:46:503
2352199211,13cyclictest0-21swapper/310:12:413
23521992111,2cyclictest25094-21diskstats12:05:153
2352199211,11cyclictest41-21ksoftirqd/311:26:353
2352199211,11cyclictest29428-21/usr/sbin/munin08:35:123
2352199210,3cyclictest41-21ksoftirqd/309:36:593
2352199210,13cyclictest39-21rcuc/311:44:493
2352199210,13cyclictest28133-21kworker/u8:011:13:303
2351699215,3cyclictest86850irq/129-enp1s0-12:15:272
23516992116,3cyclictest15816-21grep09:15:252
23516992113,5cyclictest3380-21/usr/sbin/munin07:35:162
23516992113,5cyclictest28835-21df11:00:132
2350999218,3cyclictest29691-21fschecks_time12:15:161
2350999213,2cyclictest86750irq/128-enp1s0-10:11:591
2350999212,10cyclictest12236-21diskmemload09:35:131
2350999211,8cyclictest296102irqstats07:20:201
23509992114,2cyclictest0-21swapper/110:08:561
23509992113,3cyclictest0-21swapper/108:41:031
23509992113,2cyclictest0-21swapper/111:45:381
2350999211,18cyclictest0-21swapper/108:56:411
2350999211,18cyclictest0-21swapper/107:10:251
2350999211,15cyclictest0-21swapper/112:25:121
2350999211,15cyclictest0-21swapper/109:25:091
2350999211,15cyclictest0-21swapper/109:03:481
2350999211,15cyclictest0-21swapper/108:46:161
2350999211,15cyclictest0-21swapper/108:46:161
2350999211,15cyclictest0-21swapper/107:38:481
2350999211,12cyclictest22326-21fschecks_count10:45:181
2350999211,11cyclictest0-21swapper/111:35:001
2350999211,10cyclictest0-21swapper/112:03:581
2350999211,10cyclictest0-21swapper/112:03:581
2350999210,14cyclictest0-21swapper/109:10:501
2350499219,3cyclictest11608-21idleruntime10:20:170
2350499212,12cyclictest10472-1kworker/u9:012:20:050
23504992111,3cyclictest86750irq/128-enp1s0-11:30:520
2350499211,11cyclictest7351-21kernelversion10:10:190
2350499211,11cyclictest26980-21fschecks_count07:15:160
2350499211,11cyclictest0-21swapper/009:19:040
2350499211,10cyclictest13089-21unixbench_singl11:35:250
238312200,2sleep30-21swapper/310:49:423
2352199206,3cyclictest0-21swapper/308:23:363
2352199206,3cyclictest0-21swapper/308:15:243
2352199204,5cyclictest0-21swapper/307:22:323
2352199204,3cyclictest0-21swapper/308:44:263
23521992014,3cyclictest9001-21grep07:45:283
2352199201,14cyclictest0-21swapper/312:37:033
23516992011,6cyclictest12123-21grep11:35:162
23509992013,4cyclictest28571-21munin-run07:20:011
23509992012,3cyclictest0-21swapper/109:07:541
2350999201,15cyclictest0-21swapper/111:55:211
2350499208,4cyclictest29145-21/usr/sbin/munin12:15:240
2350499201,9cyclictest0-21swapper/010:06:070
2350499201,11cyclictest7137-21/usr/sbin/munin11:25:190
2350499201,11cyclictest23704-21/usr/sbin/munin07:10:180
2350499201,10cyclictest8979-21unixbench_singl07:45:270
2350499201,10cyclictest24309-21grep08:20:260
2352199198,3cyclictest86550irq/126-enp1s0-08:50:223
2352199198,3cyclictest86550irq/126-enp1s0-08:05:213
2352199193,5cyclictest0-21swapper/311:35:363
2352199191,9cyclictest0-21swapper/311:00:163
2352199191,4cyclictest0-21swapper/310:24:533
2352199191,15cyclictest1392-21cut07:30:183
2352199191,10cyclictest0-21swapper/311:18:053
2350999198,2cyclictest28447-21ps08:30:231
2350999191,3cyclictest0-21swapper/107:56:451
23509991913,3cyclictest7788-21grep11:25:171
23509991913,2cyclictest0-21swapper/108:39:061
2350499199,2cyclictest0-21swapper/011:53:090
2350499196,3cyclictest2777-21kworker/0:107:51:130
2350499191,9cyclictest2922-21kernelversion10:00:180
2350499191,9cyclictest12812-21strings07:55:200
2350499191,9cyclictest0-21swapper/012:26:050
2350499191,8cyclictest0-21swapper/012:30:130
2350499191,8cyclictest0-21swapper/011:40:310
2350499191,3cyclictest24095-21head12:00:280
2350499191,3cyclictest24095-21head12:00:280
23504991912,4cyclictest16565-21df08:05:130
2350499191,10cyclictest14634-21snmpd08:12:490
2352199186,3cyclictest0-21swapper/312:03:033
2352199186,3cyclictest0-21swapper/312:03:033
2352199185,2cyclictest0-21swapper/307:50:363
2352199183,5cyclictest0-21swapper/308:30:043
2352199183,3cyclictest0-21swapper/308:58:203
2352199181,3cyclictest0-21swapper/312:15:073
23521991812,3cyclictest18433-21/usr/sbin/munin08:10:123
2352199180,2cyclictest0-21swapper/309:49:413
2350499187,3cyclictest24632-21ps09:35:230
2350499182,5cyclictest0-21swapper/012:11:310
2350499181,7cyclictest0-21swapper/010:25:140
2350499181,7cyclictest0-21swapper/010:15:200
2350499181,6cyclictest0-21swapper/009:34:290
172392181,2sleep00-21swapper/010:34:450
2352199171,7cyclictest0-21swapper/309:30:243
23521991712,3cyclictest25224-21awk07:10:273
2352199170,6cyclictest0-21swapper/308:01:063
2350499178,2cyclictest86750irq/128-enp1s0-10:40:500
2350499171,7cyclictest26303-21/usr/sbin/munin10:55:250
2350499171,7cyclictest0-21swapper/009:21:410
2350499171,7cyclictest0-21swapper/008:39:200
2350499171,5cyclictest0-21swapper/010:38:130
23504991712,3cyclictest8065-21acpi09:00:130
23504991712,3cyclictest27878-21cut08:30:160
2350499171,14cyclictest0-21swapper/007:25:000
23504991711,3cyclictest3724-21needreboot11:15:200
2350499171,11cyclictest0-21swapper/008:25:190
23504991710,4cyclictest6916-21/usr/sbin/munin11:25:010
159352170,2sleep00-21swapper/011:45:100
2350499161,6cyclictest0-21swapper/008:45:170
2350499161,6cyclictest0-21swapper/008:45:170
23504991612,3cyclictest0-21swapper/007:39:030
23504991611,3cyclictest24744-21/usr/sbin/munin12:05:170
23504991610,3cyclictest29463-21head07:20:180
23504991510,3cyclictest31631-21acpi08:40:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional