You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-29 - 01:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot5s.osadl.org (updated Tue Apr 28, 2026 12:44:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26152992725,1cyclictest13-21ksoftirqd/011:40:000
26152992623,2cyclictest20710-21cron10:45:000
26154992523,0cyclictest0-21swapper/210:04:502
2615499250,24cyclictest21778-21ssh10:14:052
26153992525,0cyclictest0-21swapper/109:54:321
2615399250,24cyclictest0-21swapper/111:33:051
26155992421,2cyclictest27379-21ssh11:22:003
2615599240,23cyclictest7557-21ssh10:00:363
2615599240,23cyclictest19577-21diskmemload11:01:033
2615599240,1cyclictest5103-21ssh11:31:513
26154992422,1cyclictest26982-21ssh11:53:222
26154992421,0cyclictest0-21swapper/209:36:082
26154992420,1cyclictest0-21swapper/209:50:312
2615499240,23cyclictest19577-21diskmemload10:27:122
2615499240,23cyclictest19577-21diskmemload09:26:332
2615499240,23cyclictest19577-21diskmemload09:22:182
2615499240,23cyclictest0-21swapper/211:01:242
26153992424,0cyclictest0-21swapper/110:05:061
26153992422,1cyclictest24127-21cron11:20:001
26153992421,2cyclictest28163-21sh11:23:121
2615399240,23cyclictest25450-21apt-get08:15:471
2615399240,23cyclictest0-21swapper/112:36:181
2615399240,23cyclictest0-21swapper/111:55:401
2615399240,1cyclictest0-21swapper/109:23:341
2615399240,0cyclictest0-21swapper/112:22:131
26152992423,1cyclictest19577-21diskmemload10:47:100
26152992423,1cyclictest0-21swapper/010:25:520
26152992421,2cyclictest21817-21sh10:14:080
2615299240,24cyclictest0-21swapper/011:30:450
2615299240,24cyclictest0-21swapper/010:17:430
2615299240,23cyclictest4359-21apt-get11:00:470
2615299240,23cyclictest27114-21latency_hist12:25:000
2615299240,23cyclictest22869-21munin-plugin-st11:50:010
2615299240,23cyclictest19577-21diskmemload11:09:500
2615299240,23cyclictest0-21swapper/011:26:230
2615299240,23cyclictest0-21swapper/009:59:430
2615299240,23cyclictest0-21swapper/008:05:450
2615299240,1cyclictest0-21swapper/011:41:420
2615299240,1cyclictest0-21swapper/010:07:470
2615299240,1cyclictest0-21swapper/009:53:000
2615299240,0cyclictest19577-21diskmemload09:34:410
26155992323,0cyclictest19577-21diskmemload09:26:303
26155992323,0cyclictest0-21swapper/311:46:473
26155992323,0cyclictest0-21swapper/309:38:343
26155992321,2cyclictest23367-21ssh10:46:073
26155992321,1cyclictest0-21swapper/311:41:143
26155992321,1cyclictest0-21swapper/310:50:043
2615599231,21cyclictest2467-21/usr/sbin/munin12:30:563
2615599230,23cyclictest19577-21diskmemload12:21:463
2615599230,23cyclictest0-21swapper/312:27:353
2615599230,23cyclictest0-21swapper/312:15:403
2615599230,23cyclictest0-21swapper/311:27:563
2615599230,23cyclictest0-21swapper/309:51:523
2615599230,22cyclictest6933-21idleruntime-cro10:00:003
2615599230,22cyclictest32361-21ssh11:58:493
2615599230,22cyclictest26850-21sh09:47:073
2615599230,22cyclictest19577-21diskmemload11:15:583
2615599230,22cyclictest19577-21diskmemload09:21:043
2615599230,22cyclictest19577-21diskmemload09:18:113
2615599230,22cyclictest12657-21ssh10:05:323
2615599230,22cyclictest12093-21apt-get12:10:443
2615599230,22cyclictest0-21swapper/311:10:153
2615599230,22cyclictest0-21swapper/311:05:383
2615599230,22cyclictest0-21swapper/310:55:583
2615599230,22cyclictest0-21swapper/310:40:493
2615599230,22cyclictest0-21swapper/310:26:563
26154992323,0cyclictest0-21swapper/209:10:262
26154992321,1cyclictest9732-21ssh12:38:512
26154992321,1cyclictest0-21swapper/210:40:012
26154992320,2cyclictest32366-21awk08:30:012
26154992320,2cyclictest19577-21diskmemload12:10:592
26154992320,2cyclictest19577-21diskmemload11:56:122
26154992320,0cyclictest0-21swapper/209:18:462
2615499230,23cyclictest0-21swapper/209:56:122
2615499230,22cyclictest31315-21apt-get07:20:462
2615499230,22cyclictest10102-21grep11:05:482
2615499230,22cyclictest0-21swapper/212:25:212
2615499230,1cyclictest1117-21ssh11:28:402
2615499230,1cyclictest0-21swapper/211:47:202
2615499230,0cyclictest0-21swapper/211:36:212
2615499230,0cyclictest0-21swapper/211:30:472
2615499230,0cyclictest0-21swapper/209:41:392
26153992323,0cyclictest0-21swapper/112:31:001
26153992323,0cyclictest0-21swapper/112:14:081
26153992323,0cyclictest0-21swapper/111:44:441
26153992323,0cyclictest0-21swapper/111:07:581
26153992323,0cyclictest0-21swapper/110:47:581
26153992323,0cyclictest0-21swapper/110:43:321
26153992323,0cyclictest0-21swapper/110:11:011
26153992322,1cyclictest0-21swapper/110:25:171
26153992321,1cyclictest0-21swapper/112:25:021
26153992321,1cyclictest0-21swapper/112:00:011
26153992321,1cyclictest0-21swapper/111:11:181
26153992321,1cyclictest0-21swapper/111:00:481
26153992321,1cyclictest0-21swapper/109:36:051
26153992320,2cyclictest19199-21basename11:45:481
2615399230,2cyclictest27030-21ssh09:15:541
2615399230,23cyclictest0-21swapper/112:17:201
2615399230,23cyclictest0-21swapper/109:49:091
2615399230,22cyclictest7562-21ssh10:00:371
2615399230,22cyclictest5771-21ssh10:30:241
2615399230,22cyclictest4747-21apt-get08:40:021
2615399230,22cyclictest3665-21ssh09:25:371
2615399230,22cyclictest23110-21apt-get10:15:451
2615399230,22cyclictest2049-21rm09:55:061
2615399230,0cyclictest0-21swapper/112:05:111
2615399230,0cyclictest0-21swapper/111:36:251
26152992323,0cyclictest19577-21diskmemload11:15:200
26152992323,0cyclictest19577-21diskmemload10:31:240
26152992323,0cyclictest0-21swapper/011:23:250
26152992323,0cyclictest0-21swapper/010:58:140
26152992323,0cyclictest0-21swapper/009:17:140
26152992323,0cyclictest0-21swapper/009:10:270
26152992321,1cyclictest24042-21ssh09:45:260
26152992321,1cyclictest0-21swapper/011:58:070
26152992321,1cyclictest0-21swapper/011:13:280
2615299231,1cyclictest3984-21sh12:02:010
2615299230,23cyclictest0-21swapper/012:12:270
2615299230,23cyclictest0-21swapper/010:21:330
2615299230,22cyclictest3861-21ssh12:32:430
2615299230,22cyclictest24057-21diskstats11:50:480
2615299230,22cyclictest0-21swapper/012:28:280
2615299230,22cyclictest0-21swapper/010:54:100
2615299230,22cyclictest0-21swapper/010:35:510
2615299230,1cyclictest0-21swapper/008:40:020
2615299230,0cyclictest0-21swapper/012:36:540
2615299230,0cyclictest0-21swapper/009:23:390
26155992222,0cyclictest19577-21diskmemload12:09:233
26155992222,0cyclictest19577-21diskmemload12:00:443
26155992221,1cyclictest30269-21apt08:25:463
26155992221,0cyclictest5467-21apt-get08:40:473
26155992220,1cyclictest23142-21ssh11:50:073
2615599220,21cyclictest6499-21/usr/sbin/munin10:30:483
2615599220,21cyclictest0-21swapper/310:20:453
2615599220,21cyclictest0-21swapper/307:10:443
2615599220,1cyclictest8604-21/usr/sbin/munin11:35:503
2615599220,1cyclictest19577-21diskmemload10:11:163
2615599220,1cyclictest10817-21sh09:31:013
2615599220,1cyclictest0-21swapper/310:35:463
2615599220,0cyclictest0-21swapper/309:40:263
26154992222,0cyclictest0-21swapper/211:40:152
26154992222,0cyclictest0-21swapper/210:16:262
26154992221,1cyclictest28033-21ssh11:22:582
26154992221,1cyclictest12838-21apt-get10:05:472
26154992221,1cyclictest0-21swapper/210:32:142
26154992220,2cyclictest26569-21ssh12:24:092
26154992220,1cyclictest0-21swapper/210:55:292
26154992219,2cyclictest4747-21apt-get08:40:012
2615499220,22cyclictest0-21swapper/212:15:392
2615499220,22cyclictest0-21swapper/212:00:092
2615499220,22cyclictest0-21swapper/209:47:502
2615499220,21cyclictest8695-21sh09:30:282
2615499220,21cyclictest26088-21ssh10:50:042
2615499220,21cyclictest11063-21apt-get07:45:482
2615499220,21cyclictest10830-21ssh10:35:162
2615499220,21cyclictest0-21swapper/208:25:002
2615499220,1cyclictest30927-21ssh10:22:242
2615499220,1cyclictest19577-21diskmemload11:16:422
2615499220,1cyclictest0-21swapper/212:09:302
2615499220,0cyclictest0-21swapper/210:46:392
26153992221,1cyclictest2168-21ssh10:57:411
26153992220,1cyclictest5794-21perf07:35:001
26153992220,1cyclictest13300-21sh10:36:191
2615399220,22cyclictest19577-21diskmemload09:31:571
2615399220,21cyclictest32119-21ssh11:26:251
2615399220,21cyclictest19574-21cron09:10:001
2615399220,21cyclictest0-21swapper/111:51:201
2615399220,21cyclictest0-21swapper/109:40:411
2615399220,21cyclictest0-21swapper/109:11:511
2615399220,1cyclictest19577-21diskmemload10:50:481
2615399220,1cyclictest0-21swapper/110:21:051
26152992222,0cyclictest19577-21diskmemload09:35:050
26152992222,0cyclictest19577-21diskmemload09:26:300
26152992221,1cyclictest13-21ksoftirqd/009:40:170
26152992220,1cyclictest964-21dump-pmu-power07:25:000
26152992220,1cyclictest28507-21cron07:15:000
2615299220,21cyclictest3392-21awk07:30:000
2615299220,1cyclictest19577-21diskmemload12:05:480
2615299220,0cyclictest0-21swapper/010:01:090
26155992120,1cyclictest0-21swapper/307:55:463
26155992119,1cyclictest25150-21munin-run08:15:003
2615599210,21cyclictest0-21swapper/312:35:083
2615599210,0cyclictest19577-21diskmemload09:10:363
2615599210,0cyclictest0-21swapper/307:25:473
26154992121,0cyclictest0-21swapper/211:12:022
26154992121,0cyclictest0-21swapper/207:25:472
26154992120,0cyclictest0-21swapper/208:00:462
26154992119,1cyclictest30950-21cron07:20:002
2615499210,21cyclictest0-21swapper/212:30:162
2615499210,20cyclictest8666-21apt-get07:40:462
2615499210,0cyclictest0-21swapper/208:40:462
26153992119,1cyclictest30242-21sensors_fan07:15:541
2615399210,1cyclictest0-21swapper/108:05:451
2615399210,0cyclictest0-21swapper/107:10:441
26152992119,1cyclictest12707-21apt-get08:55:460
2615299210,21cyclictest0-21swapper/009:00:470
2615299210,20cyclictest3694-21apt-get07:30:470
26155992019,1cyclictest28846-21apt-get07:15:463
2615599200,19cyclictest22737-21awk08:10:003
2615599200,0cyclictest0-21swapper/308:45:483
2615599200,0cyclictest0-21swapper/308:20:463
2615599200,0cyclictest0-21swapper/307:40:463
2615599200,0cyclictest0-21swapper/307:20:463
2615499200,20cyclictest0-21swapper/207:30:472
2615499200,0cyclictest0-21swapper/208:55:462
26153992020,0cyclictest15851-21apt-get07:55:461
26153992020,0cyclictest0-21swapper/108:25:451
26153992020,0cyclictest0-21swapper/108:00:461
26153992017,2cyclictest2344-21latency_hist08:35:011
2615399200,20cyclictest0-21swapper/108:20:461
2615399200,0cyclictest0-21swapper/107:50:451
26152992020,0cyclictest0-21swapper/008:50:470
26152992020,0cyclictest0-21swapper/008:35:450
26152992020,0cyclictest0-21swapper/007:45:480
26152992019,0cyclictest13454-21apt-get07:50:450
26152992017,2cyclictest30997-21perf07:20:000
26154991919,0cyclictest26449-21apt-get07:10:442
26154991919,0cyclictest0-21swapper/208:50:472
26154991919,0cyclictest0-21swapper/208:45:482
26154991917,1cyclictest19580-21ssh09:10:002
2615499190,18cyclictest21950irq/142-nvme0q308:10:002
26153991919,0cyclictest0-21swapper/108:55:461
26153991918,1cyclictest4769-21perf08:40:011
2615399190,0cyclictest1285-21apt-get07:25:471
26155991818,0cyclictest0-21swapper/309:05:453
26155991818,0cyclictest0-21swapper/308:55:463
26155991818,0cyclictest0-21swapper/308:35:453
26152991816,1cyclictest2337-21turbostat08:35:010
26154991717,0cyclictest0-21swapper/208:10:452
26155991616,0cyclictest6089-21apt-get07:35:453
26154991616,0cyclictest0-21swapper/209:00:472
26154991616,0cyclictest0-21swapper/207:55:462
26154991616,0cyclictest0-21swapper/207:35:452
26153991614,1cyclictest0-21swapper/108:45:231
2615399160,15cyclictest948-21cron07:25:001
2615399160,15cyclictest0-21swapper/108:10:241
2615399160,0cyclictest0-21swapper/107:40:461
26152991616,0cyclictest0-21swapper/009:05:450
26152991615,1cyclictest27878-21apt-get08:20:460
2615299160,15cyclictest25085-21chrt08:14:450
2615299160,15cyclictest13-21ksoftirqd/008:00:510
26155991513,1cyclictest886-21hddtemp_smartct08:30:493
26155991513,1cyclictest5790-21munin-run07:35:003
26155991513,1cyclictest15549-21munin-plugin-st07:55:013
26155991513,1cyclictest0-21swapper/309:05:013
2615499150,14cyclictest0-21swapper/207:50:032
2615499150,12cyclictest32669-21apt-get08:30:462
26152991515,0cyclictest0-21swapper/008:25:450
26152991513,1cyclictest27573-21munin-run08:20:000
2615299150,0cyclictest0-21swapper/008:45:560
26155991414,0cyclictest0-21swapper/308:50:043
2615599140,14cyclictest0-21swapper/308:15:333
2615599140,14cyclictest0-21swapper/307:45:183
2615599140,0cyclictest0-21swapper/308:00:273
2615499140,0cyclictest0-21swapper/208:15:362
2615399140,14cyclictest0-21swapper/107:45:101
2615399140,0cyclictest0-21swapper/109:00:011
2615399140,0cyclictest0-21swapper/107:36:151
26152991414,0cyclictest0-21swapper/007:55:420
26152991414,0cyclictest0-21swapper/007:40:290
2615299140,0cyclictest0-21swapper/007:35:040
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional