You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-22 - 03:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Wed Apr 22, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
117282240162,26sleep00-21swapper/019:05:020
137602204168,24sleep20-21swapper/219:08:172
136302202166,24sleep10-21swapper/119:06:361
135622202166,24sleep30-21swapper/319:05:453
181432570,4sleep07-21ksoftirqd/019:15:210
228292550,4sleep313963-21cyclictest21:55:133
13988995046,3cyclictest0-21swapper/220:40:062
13975994011,13cyclictest27509-21basename23:20:010
13975994010,14cyclictest27833-21munin-run19:40:020
13975993514,12cyclictest9392-21/usr/sbin/munin22:35:260
1397599345,14cyclictest10522-21munin-run23:55:020
1397599316,15cyclictest20799-21sendmail_mailqu00:15:230
1397599316,12cyclictest23116-21awk23:10:010
1397599315,19cyclictest19543-21sendmail_mailqu21:45:240
13975993112,5cyclictest6412-21hddtemp_smartct22:30:160
13975993110,10cyclictest827-21runrttasks22:06:000
1399299303,16cyclictest0-21swapper/320:24:033
1397599308,7cyclictest10854-21latency_hist20:15:000
1397599307,11cyclictest1746-21perf23:35:010
1397599307,11cyclictest1746-21perf23:35:010
13975993012,9cyclictest2770-21irqstats23:35:190
1399299291,11cyclictest0-21swapper/321:09:373
1397599298,6cyclictest32125-21expr22:15:160
1397599297,10cyclictest13908-21/usr/sbin/munin21:35:220
1397599296,10cyclictest808-21snmpd19:54:250
1397599294,12cyclictest0-21swapper/023:40:170
13975992911,4cyclictest31181-21smart_sda23:25:240
13975992911,4cyclictest24377-21ntp_kernel_pll_23:10:210
13975992910,11cyclictest16459-21diskstats21:40:140
1399299281,11cyclictest30029-21expr23:25:133
1399299281,11cyclictest0-21swapper/319:34:363
1399299281,10cyclictest0-21swapper/322:44:363
1399299280,10cyclictest0-21swapper/323:04:133
1397599289,9cyclictest28062-21df23:20:140
1397599289,11cyclictest28363-21/usr/sbin/munin00:35:250
1397599287,10cyclictest5609-21grep22:30:020
1397599286,13cyclictest6400-21cpu23:45:120
1399299274,3cyclictest0-21swapper/320:41:403
1399299271,12cyclictest0-21swapper/321:37:133
1399299271,12cyclictest0-21swapper/300:27:523
1399299270,12cyclictest0-21swapper/300:17:493
1397599279,10cyclictest12905-21/usr/sbin/munin22:45:150
1397599277,16cyclictest24218-21grep19:30:160
1397599276,13cyclictest19270-21df_inode23:00:130
1397599275,14cyclictest27228-21ntp_states00:30:210
13975992717,6cyclictest9860-21diskstats21:25:150
1399299265,4cyclictest0-21swapper/319:49:443
1399299261,9cyclictest0-21swapper/322:50:373
1399299261,12cyclictest0-21swapper/323:35:573
1399299261,12cyclictest0-21swapper/300:03:573
1399299261,11cyclictest0-21swapper/322:35:073
1399299261,11cyclictest0-21swapper/319:44:003
1399299261,11cyclictest0-21swapper/319:36:203
1399299261,10cyclictest0-21swapper/323:15:173
1399299261,10cyclictest0-21swapper/300:33:403
13992992610,3cyclictest28027-21memory22:05:203
1399299260,9cyclictest0-21swapper/321:40:173
1399299260,11cyclictest0-21swapper/321:47:293
1399299260,11cyclictest0-21swapper/320:38:083
1399299260,10cyclictest0-21swapper/323:50:413
13981992619,4cyclictest22346-21/usr/sbin/munin23:05:231
1397599269,5cyclictest87550irq/128-enp1s0-20:00:210
1397599269,14cyclictest6648-21grep21:15:270
1397599269,14cyclictest6648-21grep21:15:270
1397599269,10cyclictest22355-21fw_packets00:20:160
1397599268,14cyclictest2689-21diskmemload22:55:230
1397599266,16cyclictest23694-21grep21:55:210
1397599266,12cyclictest0-21swapper/000:05:280
1397599265,11cyclictest2461-21pmu-power22:20:230
1397599265,11cyclictest2461-21pmu-power22:20:230
1397599263,14cyclictest0-21swapper/021:20:260
1397599263,12cyclictest14529-21/usr/sbin/munin22:50:180
13975992618,6cyclictest28445-21crond20:55:000
13975992618,5cyclictest29656-21diskstats22:10:140
13975992617,5cyclictest12005-21smart_sda23:55:230
1399299256,2cyclictest0-21swapper/321:24:573
1399299255,3cyclictest0-21swapper/320:08:233
1399299255,3cyclictest0-21swapper/320:08:233
1399299251,16cyclictest0-21swapper/323:07:473
1399299251,11cyclictest0-21swapper/321:12:493
1399299251,10cyclictest0-21swapper/319:27:083
1399299251,10cyclictest0-21swapper/300:36:163
1399299251,10cyclictest0-21swapper/300:05:093
1399299250,12cyclictest0-21swapper/323:48:013
1399299250,10cyclictest0-21swapper/323:44:293
1398899253,3cyclictest5085-21sed21:15:022
13981992511,12cyclictest14965-21kworker/u8:219:16:161
1398199250,17cyclictest0-21swapper/122:24:061
1398199250,17cyclictest0-21swapper/122:24:061
1397599259,5cyclictest14380-21timerwakeupswit00:00:260
1397599258,14cyclictest2689-21diskmemload22:40:150
1397599257,15cyclictest22080-21head19:25:150
1397599257,15cyclictest20507-21/usr/sbin/munin21:50:240
1397599256,8cyclictest3734-21taskset21:10:190
1397599255,8cyclictest0-21swapper/021:00:190
1397599253,8cyclictest0-21swapper/000:25:250
1397599253,10cyclictest0-21swapper/021:30:170
13975992516,6cyclictest15443-21smart_sda19:10:240
1397599251,13cyclictest2698-21munin-run21:10:000
1399299247,3cyclictest0-21swapper/322:30:433
1399299247,11cyclictest0-21swapper/323:12:133
1399299241,9cyclictest0-21swapper/321:28:283
1399299241,9cyclictest0-21swapper/321:18:253
1399299241,9cyclictest0-21swapper/321:18:253
1399299241,9cyclictest0-21swapper/320:17:423
1399299241,10cyclictest0-21swapper/320:52:003
13992992410,3cyclictest18144-21ntp_states19:15:223
1399299240,9cyclictest0-21swapper/321:03:003
1399299240,9cyclictest0-21swapper/320:11:233
1399299240,9cyclictest0-21swapper/319:56:473
1399299240,9cyclictest0-21swapper/319:20:263
1399299240,11cyclictest0-21swapper/320:26:153
1399299240,10cyclictest0-21swapper/322:15:203
1399299240,10cyclictest0-21swapper/322:10:403
1399299240,10cyclictest0-21swapper/322:01:083
1398199241,16cyclictest0-21swapper/122:40:281
13981992411,11cyclictest24307-21kworker/u8:021:55:331
1397599249,8cyclictest827-21runrttasks20:42:070
1397599245,6cyclictest87350irq/126-enp1s0-20:20:240
13975992417,4cyclictest26007-21/usr/sbin/munin22:00:230
13975992416,5cyclictest19346-21munin-run00:15:010
13975992415,6cyclictest19270-21/usr/sbin/munin19:20:140
13975992415,5cyclictest18035-21diskstats20:30:130
150892230,4sleep11398199cyclictest22:50:121
1399299239,3cyclictest4838-21diskstats20:00:133
1399299231,2cyclictest87650irq/129-enp1s0-22:26:253
1399299231,15cyclictest0-21swapper/320:55:523
1399299231,11cyclictest0-21swapper/320:32:313
1399299231,11cyclictest0-21swapper/319:13:353
1399299231,10cyclictest32326-21ls23:30:133
1399299231,10cyclictest32326-21ls23:30:133
1399299230,9cyclictest0-21swapper/321:53:053
1399299230,9cyclictest0-21swapper/321:34:173
1399299230,10cyclictest0-21swapper/300:13:093
1398899238,6cyclictest0-21swapper/223:55:112
13988992310,7cyclictest0-21swapper/219:34:232
1398199239,12cyclictest29957-21kworker/u8:020:33:371
1398199239,12cyclictest13472-21kworker/u8:021:35:061
1398199231,19cyclictest29082-21/usr/sbin/munin22:10:011
1397599237,6cyclictest2751-21grep19:55:150
1397599236,15cyclictest25272-21grep20:45:180
1397599235,15cyclictest19885-21/usr/sbin/munin20:35:130
13975992316,4cyclictest7475-21/usr/sbin/munin20:05:190
13975992316,4cyclictest7475-21/usr/sbin/munin20:05:190
13975992315,6cyclictest28679-21/usr/sbin/munin20:55:250
13975992315,5cyclictest12154-21ntp_states20:15:210
13975992314,6cyclictest28060-21/usr/sbin/munin19:40:200
13975992311,9cyclictest17593-21cat20:30:010
1399299225,3cyclictest0-21swapper/323:57:213
1399299225,3cyclictest0-21swapper/323:20:333
1399299221,11cyclictest0-21swapper/322:23:023
1399299221,11cyclictest0-21swapper/322:23:023
1399299220,9cyclictest0-21swapper/322:49:293
13988992217,3cyclictest470-21irqstats23:30:172
13988992217,3cyclictest470-21irqstats23:30:172
1398199229,11cyclictest6811-21kworker/u8:121:49:401
1398199221,3cyclictest32227-21munin-run19:50:011
13981992213,3cyclictest0-21swapper/100:28:111
1398199221,18cyclictest32089-21processes21:00:231
1398199221,15cyclictest0-21swapper/122:45:181
13981992211,3cyclictest28129-21latency_hist00:35:011
1398199221,10cyclictest29285-21grep00:35:211
13975992213,6cyclictest30256-21/usr/sbin/munin19:45:140
93622210,3sleep10-21swapper/123:50:181
61642210,2sleep20-21swapper/220:01:162
274672210,3sleep20-21swapper/223:19:492
1399299210,10cyclictest0-21swapper/320:49:443
1398899211,10cyclictest808-21snmpd21:45:252
1398199212,6cyclictest28796-21ntp_states23:20:211
1398199212,15cyclictest5985-40dnf23:43:251
1398199211,6cyclictest25874-21df_inode23:15:141
1398199211,6cyclictest0-21swapper/123:57:221
1398199211,19cyclictest0-21swapper/121:20:531
1398199211,18cyclictest0-21swapper/120:48:261
1398199211,11cyclictest28581-21tune2fs19:40:161
1398199211,11cyclictest19260-21df23:00:121
1398199211,10cyclictest0-21swapper/119:20:111
1398199210,16cyclictest0-21swapper/100:06:281
1399299201,11cyclictest684-21hddtemp_smartct19:50:153
1399299200,11cyclictest0-21swapper/322:57:293
1398899208,3cyclictest14205-1kworker/u9:123:00:242
1398899204,3cyclictest87350irq/126-enp1s0-00:30:132
1398899202,7cyclictest3929-21munin-run23:40:002
1398899202,5cyclictest22583-21irqstats00:20:172
13988992014,3cyclictest31275-21munin-run22:15:002
1398199208,5cyclictest10501-21strings21:25:191
1398199207,11cyclictest29957-21kworker/u8:019:26:111
1398199206,12cyclictest6811-21kworker/u8:122:00:281
13981992017,2cyclictest0-21swapper/121:53:501
13981992017,2cyclictest0-21swapper/121:17:251
13981992017,2cyclictest0-21swapper/121:17:251
13981992014,4cyclictest17836-21ls22:55:231
13981992014,1cyclictest0-21swapper/100:13:581
1398199201,3cyclictest0-21swapper/122:35:141
1398199201,2cyclictest15576-21cpuspeed_turbos20:25:131
13981992012,2cyclictest0-21swapper/100:23:501
1398199201,17cyclictest0-21swapper/120:43:351
1398199201,11cyclictest2710-21fschecks_count19:55:151
1398199201,11cyclictest20832-21smart_sda00:15:241
1398199201,10cyclictest31432-21cat22:15:021
1399299191,4cyclictest0-21swapper/300:21:423
1398899192,2cyclictest87650irq/129-enp1s0-00:16:572
1398899191,8cyclictest28359-21sendmail_mailst22:05:242
1398899191,7cyclictest0-21swapper/223:28:382
1398899191,4cyclictest31869-21kworker/u8:123:48:452
13988991913,3cyclictest26451-21grep19:35:152
1398899191,10cyclictest16388-21/usr/sbin/munin00:05:242
1398199197,3cyclictest32456-21df23:30:131
1398199197,3cyclictest32456-21df23:30:121
1398199192,14cyclictest24613-21grep23:10:241
1398199191,6cyclictest579-21systemd-journal20:05:011
1398199191,5cyclictest0-21swapper/123:48:311
13981991915,3cyclictest0-21swapper/121:30:111
13981991915,2cyclictest17523-21timerwakeupswit21:40:241
1398199191,4cyclictest14538-21/usr/sbin/munin19:10:141
13981991912,2cyclictest0-21swapper/100:04:151
1398199191,17cyclictest0-21swapper/123:25:191
1398199191,17cyclictest0-21swapper/122:26:021
1398199191,17cyclictest0-21swapper/119:50:531
1398199191,16cyclictest0-21swapper/121:12:071
1398199191,16cyclictest0-21swapper/120:10:181
1398199191,11cyclictest87650irq/129-enp1s0-22:18:511
1398199191,10cyclictest6434-21ls22:30:161
1398199190,17cyclictest0-21swapper/119:30:131
304342181,2sleep20-21swapper/220:57:262
227562181,2sleep20-21swapper/223:05:452
185642181,2sleep20-21swapper/222:58:302
1398899186,4cyclictest6358-21/usr/sbin/munin21:15:252
1398899186,4cyclictest6358-21/usr/sbin/munin21:15:252
1398899186,4cyclictest2689-21diskmemload21:20:172
1398899184,5cyclictest0-21swapper/222:25:062
1398899181,7cyclictest27860-21cpuspeed_turbos23:20:132
1398899181,7cyclictest10503-21needreboot21:25:202
1398899181,7cyclictest0-21swapper/221:05:172
13988991813,4cyclictest0-21swapper/223:10:142
13988991812,3cyclictest31210-21df21:00:132
1398899181,15cyclictest2689-21diskmemload22:53:132
1398199186,11cyclictest31869-21kworker/u8:123:39:381
1398199181,3cyclictest11435-21diskstats20:15:141
1398199181,2cyclictest0-21swapper/121:06:391
1398199181,16cyclictest0-21swapper/120:56:441
1398199181,16cyclictest0-21swapper/120:51:061
1398199181,16cyclictest0-21swapper/120:36:261
1398199181,10cyclictest0-21swapper/120:20:191
1398199180,16cyclictest0-21swapper/120:05:061
1398199180,16cyclictest0-21swapper/120:05:061
321032170,2sleep20-21swapper/219:48:352
1398899172,9cyclictest0-21swapper/221:30:502
1398899171,7cyclictest7046-21diskstats20:05:162
1398899171,7cyclictest7046-21diskstats20:05:162
1398899171,7cyclictest0-21swapper/222:35:062
1398899171,6cyclictest579-21systemd-journal21:55:012
1398899171,6cyclictest24668-21latency_hist22:00:002
1398899171,6cyclictest14075-21cpuspeed_turbos21:35:132
1398899171,6cyclictest0-21swapper/222:40:192
1398899171,6cyclictest0-21swapper/222:00:142
1398899171,6cyclictest0-21swapper/221:41:372
1398899171,6cyclictest0-21swapper/200:14:392
1398899171,5cyclictest0-21swapper/220:30:242
13988991712,3cyclictest9580-21hwlatdetect20:10:172
13988991712,3cyclictest28698-21grep19:40:152
13988991712,3cyclictest20522-21gltestperf20:35:172
1398899171,15cyclictest0-21swapper/222:30:362
1398899171,14cyclictest0-21swapper/200:00:122
13988991711,3cyclictest827-21runrttasks19:54:462
1398899171,13cyclictest28477-21grep00:35:142
13988991711,3cyclictest26483-21/usr/sbin/munin20:50:182
13988991710,4cyclictest0-21swapper/219:55:252
1398199171,2cyclictest0-21swapper/119:35:301
1398899161,9cyclictest0-21swapper/219:28:072
1398899161,9cyclictest0-21swapper/200:25:082
1398899161,6cyclictest5145-21strings23:40:202
1398899161,3cyclictest2689-21diskmemload23:50:172
1398899161,3cyclictest0-21swapper/222:23:072
1398899161,3cyclictest0-21swapper/222:23:062
13988991612,3cyclictest0-21swapper/220:29:022
1398899161,14cyclictest0-21swapper/220:18:322
13988991611,3cyclictest31502-21/usr/sbin/munin22:15:212
13988991611,3cyclictest19741-21grep19:20:152
13988991611,3cyclictest12325-21/usr/sbin/munin22:45:152
1398899160,15cyclictest0-21swapper/220:49:262
1398899153,10cyclictest14815-21head19:10:152
13988991510,3cyclictest17493-21grep19:15:162
1398899149,3cyclictest13287-21acpi20:20:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional