You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-10-20 - 00:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Mon Oct 19, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
100821140,1sleep30-21swapper/310:47:193
705429985,9sleep00-21swapper/007:09:520
703229467,9sleep30-21swapper/307:09:353
681429178,9sleep20-21swapper/207:06:452
691528966,13sleep10-21swapper/107:08:041
716099271,1cyclictest10584-21ssh09:35:193
715399240,1cyclictest0-21swapper/010:21:310
7155992322,1cyclictest0-21swapper/112:08:251
7155992321,2cyclictest25-21ksoftirqd/111:14:091
7158992221,1cyclictest0-21swapper/212:26:082
7155992221,1cyclictest0-21swapper/110:39:211
152922220,0sleep232-21ksoftirqd/212:13:542
7158992110,6cyclictest25454-21apt-get09:25:132
715599200,3cyclictest0-21swapper/112:13:071
715599200,0cyclictest0-21swapper/110:53:311
715399201,18cyclictest30192-21apache_processe11:05:130
715899190,19cyclictest0-21swapper/210:08:232
715599192,16cyclictest31590-21systemctl09:05:221
7155991918,1cyclictest0-21swapper/112:25:131
715599191,18cyclictest49-21kcompactd010:34:251
715599190,1cyclictest0-21swapper/111:01:121
715399191,16cyclictest0-21swapper/010:10:130
715899180,1cyclictest0-21swapper/209:48:392
715899180,17cyclictest0-21swapper/210:56:122
715399186,6cyclictest18413-21tr09:59:590
7153991814,3cyclictest26602-21sh11:02:450
7160991710,6cyclictest8300-21sh11:10:363
715899176,9cyclictest9604-21apt-get09:35:132
715899170,16cyclictest0-21swapper/211:38:192
715599170,17cyclictest0-21swapper/109:48:561
715399174,6cyclictest25012-21idleruntime-cro08:55:010
716099162,10cyclictest22280-21run-parts11:20:003
716099161,14cyclictest12667-21kworker/u8:012:23:043
715899160,2cyclictest0-21swapper/211:56:082
715899160,1cyclictest0-21swapper/209:20:022
715899160,16cyclictest0-21swapper/212:18:402
715899160,15cyclictest0-21swapper/210:41:542
715599163,4cyclictest273-21systemd-udevd12:22:041
715599163,2cyclictest0-21swapper/109:50:141
715599160,1cyclictest0-21swapper/109:39:051
715599160,16cyclictest0-21swapper/111:58:571
715399162,8cyclictest20278-21awk08:45:000
715399160,1cyclictest141rcu_preempt10:58:190
206592160,1chrt20663-21gltestperf10:20:171
715899150,1cyclictest0-21swapper/212:21:292
715899150,14cyclictest21677-21perf07:40:002
715899150,13cyclictest0-21swapper/212:08:252
7155991514,0cyclictest0-21swapper/111:44:131
7155991510,3cyclictest9285-21taskset07:12:281
715399153,6cyclictest9562-21idleruntime-cro12:29:590
7160991410,3cyclictest31170-21cron07:59:593
7160991410,3cyclictest2605-21cron10:10:003
7160991410,2cyclictest13544-21tr08:30:143
716099140,13cyclictest20476-21sed12:35:283
7158991413,1cyclictest0-21swapper/209:55:232
7158991410,3cyclictest15928-21mailstats11:15:222
715899140,14cyclictest0-21swapper/211:53:402
715899140,14cyclictest0-21swapper/211:53:392
715899140,14cyclictest0-21swapper/209:40:122
715599141,1cyclictest0-21swapper/109:14:531
7155991411,2cyclictest29620-21df07:55:151
7155991410,3cyclictest2647-21cat10:09:591
7155991410,2cyclictest14482-21dump-pmu-power07:25:001
715599140,1cyclictest0-21swapper/110:19:191
715599140,1cyclictest0-21swapper/110:19:181
7153991413,1cyclictest0-21swapper/011:57:230
7153991411,2cyclictest10963-21wc07:15:190
716099139,3cyclictest9684-21grep12:10:153
716099139,3cyclictest8577-21sed11:50:153
716099139,3cyclictest8577-21sed11:50:143
716099139,3cyclictest4421-21cut08:10:163
716099139,3cyclictest3545-21grep12:25:253
716099134,4cyclictest10032-21apache_processe12:30:113
7160991311,1cyclictest0-21swapper/311:26:333
7160991310,2cyclictest7280-21tr11:30:133
7160991310,2cyclictest23519-21grep11:00:253
7160991310,2cyclictest19298-21cron10:20:013
7160991310,2cyclictest19298-21cron10:20:003
7160991310,2cyclictest17877-21turbostat.cron08:40:003
716099130,1cyclictest0-21swapper/310:22:493
7158991312,1cyclictest0-21swapper/211:43:052
7158991310,2cyclictest7836-21grep08:15:272
7158991310,2cyclictest7476-21basename07:10:122
7158991310,2cyclictest4646-21cut10:10:222
7158991310,2cyclictest27458-21sed07:50:192
7158991310,2cyclictest20623-21cat10:00:252
715899130,1cyclictest0-21swapper/209:17:462
715899130,13cyclictest0-21swapper/211:24:222
715899130,13cyclictest0-21swapper/210:20:182
715599139,3cyclictest25495-21ssh12:39:221
715599139,3cyclictest15871-21sort11:55:011
715599139,3cyclictest15871-21sort11:55:001
715599138,2cyclictest21218-21tune2fs08:45:181
715599131,7cyclictest38150irq/131-snd_hda08:40:131
7155991312,1cyclictest0-21swapper/112:31:561
7155991312,1cyclictest0-21swapper/111:29:511
7155991311,2cyclictest25-21ksoftirqd/111:31:351
7155991310,2cyclictest23020-21basename08:50:151
7155991310,2cyclictest21498-21grep10:40:171
7155991310,2cyclictest19089-21grep09:20:271
7155991310,2cyclictest17907-21tail07:30:171
7155991310,2cyclictest17377-21grep08:35:251
7155991310,2cyclictest12219-21ssh09:55:271
715599130,8cyclictest0-21swapper/111:16:471
715599130,13cyclictest0-21swapper/109:28:251
715599130,10cyclictest28299-21basename10:25:151
7153991310,3cyclictest11843-21ls10:35:010
7153991310,2cyclictest7401-21tr08:15:220
7153991310,2cyclictest693-21grep12:05:150
7153991310,2cyclictest31962-21grep11:25:180
7153991310,2cyclictest30731-21tail07:55:270
7153991310,2cyclictest24661-21grep11:20:260
7153991310,2cyclictest1702-21grep09:10:200
7153991310,2cyclictest1646-21tr08:05:140
7153991310,2cyclictest15211-21cut11:15:170
7153991310,2cyclictest12525-21cut10:15:200
7153991310,2cyclictest12525-21cut10:15:190
715399130,2cyclictest0-21swapper/009:05:130
715399130,12cyclictest30415-21sed10:45:220
716099129,2cyclictest5777-21tail10:50:163
716099129,2cyclictest4106-21basename10:30:133
716099129,2cyclictest25146-21ssh11:40:203
716099129,2cyclictest21125-21sort08:45:193
716099129,2cyclictest20182-21sort07:35:163
716099129,2cyclictest16319-21grep07:25:263
716099129,2cyclictest12448-21basename10:35:133
716099128,3cyclictest17219-21grep11:55:183
716099128,2cyclictest0-21swapper/311:06:093
716099124,5cyclictest8314-21basename08:20:003
716099122,9cyclictest0-21swapper/311:35:163
7160991210,2cyclictest39-21ksoftirqd/309:33:003
715899129,3cyclictest0-21swapper/210:35:152
715899129,2cyclictest448-21head11:45:172
715899129,2cyclictest448-21head11:45:162
715899129,2cyclictest32-21ksoftirqd/207:40:192
715899129,2cyclictest28761-21taskset10:45:012
715899129,2cyclictest25516-21grep08:55:162
715899129,2cyclictest25516-21grep08:55:152
715899129,2cyclictest23606-21grep08:50:172
715899129,2cyclictest16914-21sh07:30:012
715899126,3cyclictest0-21swapper/207:59:592
715899121,9cyclictest32369-21diskmemload12:32:212
7158991210,1cyclictest0-21swapper/210:17:402
7158991210,1cyclictest0-21swapper/210:17:392
715899120,11cyclictest16815-21tr08:35:202
715599129,2cyclictest660-21grep08:00:291
715599129,2cyclictest28836-21cat09:00:211
715599129,2cyclictest22933-21tr11:20:141
715599129,2cyclictest22726-21grep07:40:171
715599129,2cyclictest2141-21head09:30:171
715599129,2cyclictest18755-21head09:40:151
715599129,2cyclictest17541-21grep12:15:151
715599128,3cyclictest26134-21taskset07:46:161
715599127,4cyclictest21378-21ssh11:38:321
715599127,4cyclictest18723-21ssh10:58:081
715599127,4cyclictest15788-21ssh09:19:231
715399129,2cyclictest8793-21tail11:50:170
715399129,2cyclictest8793-21tail11:50:160
715399129,2cyclictest7797-21cat11:10:230
715399129,2cyclictest4928-21tr08:10:190
715399129,2cyclictest3746-21find09:50:230
715399129,2cyclictest27323-21grep09:25:260
715399129,2cyclictest26054-21grep12:00:200
715399129,2cyclictest19658-21sort10:00:160
715399129,2cyclictest17424-21cron09:40:000
715399129,2cyclictest16070-21cut11:35:150
715399129,2cyclictest13554-21grep08:30:150
715399129,2cyclictest13051-21cut07:20:160
715399129,2cyclictest1301-21ls12:25:010
715399129,2cyclictest10697-21munin-plugin-st08:24:590
715399128,3cyclictest9048-21tail07:10:240
715399128,3cyclictest28122-21cut09:00:160
715399128,2cyclictest16593-21awk09:20:000
715399125,3cyclictest225-21systemd-journal12:35:000
715399123,8cyclictest19309-21idleruntime-cro07:35:010
715399121,1cyclictest32369-21diskmemload12:16:350
716099118,2cyclictest8535-21chrt09:15:023
716099118,2cyclictest7523-21tr07:10:133
716099118,2cyclictest6607-21ssh11:49:213
716099118,2cyclictest6607-21ssh11:49:203
716099118,2cyclictest5819-21ssh10:10:593
716099118,2cyclictest3604-21ssh12:06:153
716099118,2cyclictest3380-21memory09:50:193
716099118,2cyclictest32065-21ssh09:48:233
716099118,2cyclictest29656-21chrt09:00:523
716099118,2cyclictest28190-21ssh10:25:153
716099118,2cyclictest27875-21ssh12:01:223
716099118,2cyclictest26253-21taskset07:47:493
716099118,2cyclictest25615-21grep09:25:133
716099118,2cyclictest23588-21tail07:40:253
716099118,2cyclictest23571-21tr11:20:173
716099118,2cyclictest22302-21ssh09:22:563
716099118,2cyclictest21061-21rm10:00:333
716099118,2cyclictest20613-21ssh10:40:093
716099118,2cyclictest19358-21grep09:40:203
716099118,2cyclictest13852-21ssh10:55:133
716099118,2cyclictest12568-21grep08:25:263
716099118,2cyclictest11880-21ssh09:55:233
716099117,2cyclictest455-21dbus-daemon08:20:233
716099116,4cyclictest12078-21cron07:20:013
716099114,5cyclictest25270-21apache_processe08:55:143
716099114,5cyclictest25270-21apache_processe08:55:133
716099113,4cyclictest225-21systemd-journal08:05:003
716099112,2cyclictest225-21systemd-journal12:18:323
715899119,1cyclictest1734-21ntpq09:10:202
715899118,2cyclictest9249-21cat09:35:002
715899118,2cyclictest8614-21tr08:20:152
715899118,2cyclictest30656-21cat11:05:142
715899118,2cyclictest2729-21expr09:50:162
715899118,2cyclictest23906-21ssh11:00:382
715899118,2cyclictest22432-21ssh12:36:582
715899118,2cyclictest21238-21grep08:45:192
715899118,2cyclictest20261-21dump-pmu-power08:45:002
715899118,2cyclictest17946-21head07:30:162
715899115,5cyclictest11695-21sed07:15:262
715599119,2cyclictest25-21ksoftirqd/111:06:331
715599119,1cyclictest15430-21munin-plugin-st08:35:001
715599118,2cyclictest3126-21grep08:05:261
715599118,2cyclictest26366-21cat08:55:211
715599118,2cyclictest26366-21cat08:55:201
715599118,2cyclictest22296-21ssh10:01:321
715599118,2cyclictest21832-21wc07:40:001
715599118,2cyclictest11669-21head08:25:171
715599113,7cyclictest1-21systemd10:10:371
715599110,10cyclictest32387-21tr11:45:141
715599110,10cyclictest32387-21tr11:45:131
715399119,1cyclictest8903-21sed11:30:250
715399118,2cyclictest939-21ssh11:45:210
715399118,2cyclictest939-21ssh11:45:200
715399118,2cyclictest6861-21grep10:50:240
715399118,2cyclictest3591-21ssh09:30:400
715399118,2cyclictest32691-21tr08:00:230
715399118,2cyclictest32175-21ssh10:27:020
715399118,2cyclictest29322-21sh09:46:140
715399118,2cyclictest28454-21tail10:05:190
715399118,2cyclictest23499-21taskset11:40:030
715399118,2cyclictest22467-21chrt08:48:030
715399118,2cyclictest20535-21rm10:40:050
715399118,2cyclictest18761-21head09:40:150
715399118,2cyclictest18257-21chrt12:35:030
715399118,2cyclictest17894-21cut09:20:180
715399118,2cyclictest15531-21cut07:25:170
715399118,2cyclictest13932-21ssh10:35:230
715399117,3cyclictest25080-21cut07:45:150
716099108,1cyclictest3314-21taskset08:07:203
716099108,1cyclictest32616-21ssh09:10:023
716099108,1cyclictest32287-21chrt09:09:103
716099108,1cyclictest28029-21sendmail_mailst07:50:243
716099108,1cyclictest20059-21taskset08:42:353
716099108,1cyclictest19130-21chrt07:33:053
716099108,1cyclictest13686-21gzip07:20:223
716099107,2cyclictest22943-21cut08:50:143
715899108,2cyclictest9587-21ssh10:33:152
715899108,2cyclictest30746-21sh10:25:552
715899108,2cyclictest30656-21expr09:05:152
715899108,2cyclictest11229-21ssh10:53:492
715899108,1cyclictest7765-21tune2fs11:30:152
715899108,1cyclictest4074-21tr08:10:152
715899108,1cyclictest24547-21ssh12:00:082
715899108,1cyclictest12464-21grep07:20:132
715899107,2cyclictest5823-21ssh11:10:012
715899107,2cyclictest4796-21ssh11:28:392
715899107,2cyclictest24551-21expr07:45:122
715899107,2cyclictest13519-21sort08:30:142
715899107,2cyclictest10906-21seq08:25:042
715599108,1cyclictest6000-21cat08:14:591
715599108,1cyclictest26757-21grep07:50:161
715599108,1cyclictest15586-21if_enp0s31f607:25:181
715599107,2cyclictest6956-21tail08:15:171
715599107,2cyclictest25510-21cut12:00:151
715599107,2cyclictest10783-21cat08:24:591
715599105,2cyclictest225-21systemd-journal07:15:251
715599100,9cyclictest29200-21wc10:45:131
715399108,1cyclictest28616-21seq07:52:420
715399108,1cyclictest26221-21cut08:55:200
715399108,1cyclictest26221-21cut08:55:190
715399107,2cyclictest17889-21tr08:39:590
715399107,2cyclictest12045-21sed08:25:200
715399100,9cyclictest10572-21cat12:10:240
71589998,1cyclictest29134-21ps09:00:242
71589998,1cyclictest1839-21cat08:05:142
71589997,1cyclictest314342sleep208:00:142
71539997,1cyclictest21984-21wget07:40:140
71539996,2cyclictest19827-21tr07:35:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional