You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-04 - 07:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sat Apr 04, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
165622241165,24sleep10-21swapper/119:07:471
166732240163,25sleep30-21swapper/319:09:153
165212234162,59sleep00-21swapper/019:07:130
167222228154,23sleep20-21swapper/219:09:542
1925221810,5sleep21683099cyclictest19:15:022
2590121500,5sleep21683099cyclictest21:55:142
1928321240,5sleep21683099cyclictest21:40:142
16830997168,2cyclictest0-21swapper/223:56:012
55172490,3sleep00-21swapper/022:20:240
24572490,1sleep30-21swapper/321:00:213
1683599361,27cyclictest0-21swapper/323:30:183
1683599341,25cyclictest0-21swapper/323:17:103
1683599341,25cyclictest0-21swapper/323:17:103
1683599331,24cyclictest0-21swapper/322:53:073
1683599322,22cyclictest0-21swapper/321:52:453
1682599315,17cyclictest0-21swapper/100:22:491
1682599311,11cyclictest0-21swapper/121:55:411
1682599307,12cyclictest6141-21latency_hist22:25:011
1682599301,19cyclictest0-21swapper/121:10:181
1683599291,19cyclictest0-21swapper/323:35:203
1682599299,5cyclictest23398-21strings00:15:191
1682599291,15cyclictest0-21swapper/122:50:541
1682599290,5cyclictest0-21swapper/119:30:111
1682599287,10cyclictest0-21swapper/121:15:551
1682599281,6cyclictest0-21swapper/100:30:241
1682599281,16cyclictest0-21swapper/123:41:451
1682599280,19cyclictest0-21swapper/122:00:221
1683599274,16cyclictest87350irq/126-enp1s0-21:10:243
1682599274,18cyclictest0-21swapper/121:21:421
16825992720,4cyclictest20260-21diskstats19:15:131
1682599271,16cyclictest0-21swapper/100:06:141
1682599271,15cyclictest0-21swapper/121:44:181
1683599267,3cyclictest0-21swapper/323:28:213
1683599261,5cyclictest8768-21irqstats21:15:183
1683599261,5cyclictest14497-21grep22:40:253
1683599261,17cyclictest0-21swapper/321:49:143
1683599261,17cyclictest0-21swapper/321:36:343
1683599261,16cyclictest0-21swapper/323:09:083
1683599261,16cyclictest0-21swapper/322:26:293
1682599263,14cyclictest23-21ksoftirqd/123:20:151
1682599261,9cyclictest0-21swapper/121:29:361
1682599261,16cyclictest7431-21/usr/sbin/munin22:25:191
1682599260,14cyclictest0-21swapper/122:59:231
141432261,3sleep014270-21df_inode20:15:120
264992250,4sleep3401ktimersoftd/323:10:123
1683599258,9cyclictest0-21swapper/322:19:563
1683599258,10cyclictest0-21swapper/323:43:313
1683599257,10cyclictest0-21swapper/322:13:313
16835992518,4cyclictest0-21swapper/322:36:523
1683599251,17cyclictest0-21swapper/323:53:403
1683599251,17cyclictest0-21swapper/300:25:283
1683599251,16cyclictest808-21snmpd19:14:433
1683599251,16cyclictest0-21swapper/322:57:393
1683599251,16cyclictest0-21swapper/300:11:413
16835992510,8cyclictest10556-21awk22:35:013
1683599250,7cyclictest0-21swapper/321:43:433
1682599259,3cyclictest29428-21ntp_states23:15:201
1682599259,3cyclictest29428-21ntp_states23:15:201
1682599257,3cyclictest18411-21ls20:25:121
1682599251,9cyclictest0-21swapper/100:39:021
16825992519,3cyclictest31248-21diskstats19:40:141
1682599251,4cyclictest0-21swapper/122:30:031
16825992514,10cyclictest0-21swapper/120:18:341
1682599251,16cyclictest0-21swapper/123:31:261
1682599251,16cyclictest0-21swapper/122:39:201
1682599251,12cyclictest0-21swapper/121:30:231
1682599251,10cyclictest0-21swapper/100:13:511
1682599250,16cyclictest0-21swapper/122:45:201
1682599250,13cyclictest0-21swapper/121:53:321
1681899251,12cyclictest11171-21awk23:50:010
1683599248,9cyclictest0-21swapper/323:03:213
1683599248,9cyclictest0-21swapper/321:25:293
1683599241,15cyclictest0-21swapper/323:55:503
1683599241,10cyclictest5689-21tail22:20:263
1683099245,10cyclictest16529-21awk21:35:022
1683099240,10cyclictest0-21swapper/223:24:122
1682599249,8cyclictest0-21swapper/123:05:201
1682599248,3cyclictest5526-21dump-pmu-power21:10:021
16825992418,3cyclictest10000-21irqstats23:45:171
1682599241,20cyclictest21753-21gunzip20:30:221
1682599241,17cyclictest0-21swapper/123:36:071
1682599241,16cyclictest0-21swapper/121:36:431
1682599241,11cyclictest0-21swapper/122:08:551
1682599241,11cyclictest0-21swapper/120:44:221
1682599241,10cyclictest15931-21cpuspeed_turbos00:00:121
1682599241,10cyclictest0-21swapper/119:20:241
1682599240,15cyclictest0-21swapper/119:36:131
1683599238,3cyclictest16121-21/usr/sbin/munin20:20:173
1683599236,8cyclictest87350irq/126-enp1s0-20:10:133
16835992317,3cyclictest29313-21/usr/sbin/munin20:50:203
16835992316,5cyclictest5528-21diskmemload00:07:153
1683599231,5cyclictest0-21swapper/321:24:353
1683599231,5cyclictest0-21swapper/300:03:193
1683599231,15cyclictest0-21swapper/319:30:243
1683599231,14cyclictest0-21swapper/322:05:123
1683599231,13cyclictest0-21swapper/323:23:193
1683599231,13cyclictest0-21swapper/321:31:203
1683599231,12cyclictest0-21swapper/322:47:253
1683599231,12cyclictest0-21swapper/300:32:003
1683599231,12cyclictest0-21swapper/300:20:573
1683599231,10cyclictest24382-21munin-run00:20:023
1683599230,6cyclictest0-21swapper/323:47:113
1683099237,9cyclictest0-21swapper/222:15:182
1683099237,3cyclictest12050irq/123-ahci[0023:15:012
16825992319,3cyclictest0-21swapper/123:55:301
16825992319,3cyclictest0-21swapper/122:12:001
1682599231,18cyclictest1-21systemd21:00:021
1682599231,14cyclictest3779-21runrttasks22:18:001
1682599231,12cyclictest18995-21awk19:15:011
1682599231,10cyclictest27136-21df00:25:131
1682599231,10cyclictest14019-21awk22:40:201
1681899232,10cyclictest24783-21irqstats23:05:180
1681899231,11cyclictest0-21swapper/022:50:010
1683599225,7cyclictest755-21rngd19:24:083
1683599222,9cyclictest31199-21/usr/sbin/munin00:35:163
1683599222,15cyclictest41-21ksoftirqd/320:17:293
1683599222,10cyclictest808-21snmpd22:00:133
1683599221,4cyclictest0-21swapper/319:25:233
1683099225,3cyclictest12050irq/123-ahci[0023:50:022
1682599229,8cyclictest0-21swapper/119:55:211
1682599228,8cyclictest0-21swapper/123:51:401
16825992219,2cyclictest0-21swapper/123:10:461
16825992219,2cyclictest0-21swapper/120:20:371
16825992219,2cyclictest0-21swapper/119:27:131
1682599221,11cyclictest0-21swapper/123:25:241
1682599221,11cyclictest0-21swapper/123:00:281
1682599221,10cyclictest0-21swapper/121:47:541
1682599221,10cyclictest0-21swapper/120:45:111
1682599221,10cyclictest0-21swapper/120:00:261
1682599220,15cyclictest0-21swapper/120:35:181
1681899228,3cyclictest1748-21latency_hist22:15:010
1681899224,15cyclictest26278-21munin-run19:30:000
1681899221,5cyclictest21714-21wc23:00:020
1681899221,2cyclictest87550irq/128-enp1s0-23:50:190
118052221,3sleep211831-21ntp_states22:35:202
1683599217,3cyclictest18325-21/usr/sbin/munin20:25:223
1683599213,9cyclictest87350irq/126-enp1s0-20:46:143
1683599213,9cyclictest41-21ksoftirqd/319:37:503
1683599213,9cyclictest39-21rcuc/319:58:553
1683599213,9cyclictest0-21swapper/321:05:253
1683599211,9cyclictest0-21swapper/321:59:213
1683599211,10cyclictest0-21swapper/319:15:223
1683599210,15cyclictest0-21swapper/320:35:213
1683099217,6cyclictest30234-21sendmail_mailqu00:30:222
1683099216,7cyclictest87350irq/126-enp1s0-00:05:222
1683099214,3cyclictest28171-21ntpq20:45:192
1683099213,7cyclictest0-21swapper/223:27:262
1683099213,3cyclictest311ktimersoftd/221:30:022
1683099212,15cyclictest20038-21sshd22:55:162
16830992118,2cyclictest0-21swapper/200:26:252
1683099210,11cyclictest32214-21idleruntime20:55:152
16825992117,3cyclictest0-21swapper/120:13:441
16825992116,3cyclictest4003-21ntp_states19:50:211
1682599210,9cyclictest0-21swapper/120:05:171
1681899211,11cyclictest19228-21grep00:05:230
16835992013,4cyclictest25029-21tr20:40:113
1683599201,17cyclictest0-21swapper/320:09:513
1683599201,17cyclictest0-21swapper/319:45:173
1683599201,17cyclictest0-21swapper/319:45:173
1683099209,3cyclictest87350irq/126-enp1s0-21:20:162
1683099209,2cyclictest87350irq/126-enp1s0-21:35:332
1683099203,7cyclictest311ktimersoftd/222:50:212
1683099200,8cyclictest0-21swapper/223:04:532
1682599208,1cyclictest0-21swapper/120:50:241
16825992017,2cyclictest0-21swapper/119:47:261
16825992017,2cyclictest0-21swapper/119:47:251
1681899207,10cyclictest11-21rcuc/021:00:020
1681899206,3cyclictest87550irq/128-enp1s0-00:20:250
1681899206,3cyclictest10409-21kworker/u8:123:15:240
1681899206,3cyclictest10409-21kworker/u8:123:15:240
1681899205,6cyclictest87550irq/128-enp1s0-23:55:200
1681899204,8cyclictest87550irq/128-enp1s0-23:20:120
1681899204,8cyclictest87550irq/128-enp1s0-21:20:130
1681899202,2cyclictest87350irq/126-enp1s0-23:38:170
1681899201,4cyclictest0-21swapper/000:33:060
1681899201,3cyclictest9312-21munin-run20:05:010
1681899201,17cyclictest0-21swapper/019:50:240
1681899201,11cyclictest23340-21/usr/sbin/munin21:50:130
327582191,2sleep00-21swapper/019:43:460
241362190,3sleep00-21swapper/000:17:040
1683599191,3cyclictest12050irq/123-ahci[0020:05:013
1683099198,3cyclictest87350irq/126-enp1s0-19:25:132
1683099196,5cyclictest87650irq/129-enp1s0-22:00:222
1683099196,5cyclictest311ktimersoftd/221:55:022
1683099194,7cyclictest87350irq/126-enp1s0-19:45:182
1683099194,7cyclictest87350irq/126-enp1s0-19:45:182
1683099192,9cyclictest311ktimersoftd/223:05:162
1683099192,8cyclictest311ktimersoftd/223:50:182
1683099192,8cyclictest0-21swapper/222:24:492
1683099191,8cyclictest21652-21cat00:10:222
1683099191,6cyclictest0-21swapper/220:20:182
1683099191,5cyclictest0-21swapper/223:15:262
1683099191,5cyclictest0-21swapper/223:15:252
1683099191,10cyclictest0-21swapper/221:10:242
1682599197,2cyclictest0-21swapper/121:02:001
1681899198,3cyclictest9536-21/usr/sbin/munin20:05:220
1681899194,8cyclictest87550irq/128-enp1s0-21:35:110
1681899194,3cyclictest28149-21fschecks_count22:00:140
1681899191,9cyclictest0-21swapper/022:05:150
1681899191,8cyclictest0-21swapper/022:15:390
1681899191,7cyclictest23825-21latency_hist23:05:010
1681899191,6cyclictest26508-21diskstats23:10:120
1681899191,6cyclictest18286-21missed_timers22:50:200
1681899191,6cyclictest0-21swapper/022:25:410
1681899191,5cyclictest27145-21df_inode00:25:140
1681899191,5cyclictest0-21swapper/022:36:050
16818991913,3cyclictest16815-21/usr/sbin/munin20:20:160
1681899191,11cyclictest0-21swapper/021:26:140
1681899191,10cyclictest5851-21idleruntime19:55:150
1681899191,10cyclictest32554-21sed00:35:260
1681899191,10cyclictest25837-21expr21:55:120
1681899191,10cyclictest23588-21grep19:20:270
1681899191,10cyclictest0-21swapper/021:15:170
65992181,2sleep20-21swapper/223:38:022
1683599181,5cyclictest30982-21sa119:40:123
1683599181,5cyclictest20527-21/usr/sbin/munin20:30:253
16835991811,4cyclictest3587-21gltestperf19:50:163
1683099188,3cyclictest87650irq/129-enp1s0-19:35:242
1683099187,4cyclictest311ktimersoftd/221:05:012
1683099187,3cyclictest12050irq/123-ahci[0000:40:012
1683099185,3cyclictest32-21ksoftirqd/220:55:012
1683099184,4cyclictest9930-21perf21:20:012
1683099184,4cyclictest0-21swapper/219:20:002
1683099181,7cyclictest0-21swapper/223:30:132
1683099181,6cyclictest0-21swapper/222:05:142
1683099181,6cyclictest0-21swapper/221:48:562
1683099181,6cyclictest0-21swapper/200:20:222
1683099181,5cyclictest0-21swapper/222:26:462
1683099181,15cyclictest0-21swapper/220:15:172
1683099181,11cyclictest0-21swapper/222:31:272
1683099181,10cyclictest0-21swapper/220:30:132
1683099180,2cyclictest13781-21kernelversion22:40:182
1681899185,3cyclictest87550irq/128-enp1s0-23:40:120
1681899184,7cyclictest2599-21/usr/sbin/munin23:30:170
1681899184,4cyclictest909-21sssd_nss22:40:240
1681899184,3cyclictest11350-21kworker/0:220:10:130
1681899183,8cyclictest17836-21kworker/0:220:26:460
1681899183,4cyclictest27115-21/usr/sbin/munin20:45:170
1681899182,9cyclictest4368-21irqstats21:05:180
1681899182,10cyclictest18654-21kworker/u8:221:45:450
1681899181,9cyclictest9485-21/usr/sbin/munin22:30:190
1681899181,8cyclictest0-21swapper/019:49:390
1681899181,8cyclictest0-21swapper/019:49:390
1681899181,6cyclictest15933-21cpuspeed_turbos00:00:120
1681899181,6cyclictest0-21swapper/023:26:040
1681899181,5cyclictest21307-21open_inodes00:10:190
1681899181,5cyclictest16132-21users21:30:260
1681899181,4cyclictest0-21swapper/021:40:360
1681899181,3cyclictest18529-21sed19:10:250
16818991812,4cyclictest28464-21idleruntime-cro19:35:000
1681899181,14cyclictest25021-21grep20:40:120
125292180,5sleep21683099cyclictest20:10:172
1683099179,5cyclictest0-21swapper/219:50:162
1683099174,2cyclictest87550irq/128-enp1s0-00:00:122
1683099174,2cyclictest30-21rcuc/219:55:222
1683099172,7cyclictest311ktimersoftd/200:15:122
1683099171,8cyclictest0-21swapper/223:40:122
1683099171,6cyclictest671-21needreboot22:10:192
1683099171,3cyclictest0-21swapper/220:01:492
1683099171,15cyclictest0-21swapper/219:30:552
1683099171,14cyclictest0-21swapper/222:45:232
1683099171,14cyclictest0-21swapper/220:09:492
16830991711,3cyclictest22747-21gltestperf19:20:162
1683099170,5cyclictest30-21rcuc/220:25:122
1683099170,3cyclictest5526-21ls21:10:012
1683099170,15cyclictest0-21swapper/220:35:142
1681899174,7cyclictest5754-21/usr/sbin/munin21:10:110
1681899173,12cyclictest11-21rcuc/019:35:240
1681899171,3cyclictest24237-21sed20:35:260
1681899170,5cyclictest0-21swapper/020:50:240
1683599161,5cyclictest0-21swapper/320:55:203
16830991612,3cyclictest0-21swapper/219:41:072
1683099161,14cyclictest0-21swapper/220:44:152
16818991611,3cyclictest20527-21/usr/sbin/munin20:30:160
16818991611,3cyclictest1710-21forks21:00:150
1681899153,3cyclictest87550irq/128-enp1s0-19:15:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional