You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-15 - 11:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Dec 15, 2025 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
131502204167,24sleep30-21swapper/319:08:543
130962203166,24sleep10-21swapper/119:08:121
130132202166,24sleep20-21swapper/219:07:082
129282201164,25sleep00-21swapper/019:06:010
2419821210,3sleep01332199cyclictest23:10:260
13327997566,3cyclictest0-21swapper/123:47:121
13327997566,3cyclictest0-21swapper/123:47:121
116492560,5sleep230-21rcuc/223:55:272
72712540,1sleep20-21swapper/222:35:122
13327995047,2cyclictest0-21swapper/121:55:081
90072380,3sleep121-21rcuc/121:25:121
1332199378,13cyclictest10202-21munin-run20:15:010
1332199378,13cyclictest10202-21munin-run20:15:010
1332799321,14cyclictest0-21swapper/120:25:151
1332199325,12cyclictest6436-21basename21:20:010
1332199315,12cyclictest13029-21awk21:35:010
1332799301,11cyclictest0-21swapper/122:02:271
13334992911,3cyclictest1424-21tr23:35:122
1332799295,14cyclictest594-21smart_sda19:50:231
1332799291,12cyclictest0-21swapper/119:43:191
1332799281,11cyclictest0-21swapper/120:14:391
1332799281,11cyclictest0-21swapper/120:14:391
1332799271,11cyclictest0-21swapper/123:05:191
1332799271,11cyclictest0-21swapper/120:36:471
1332799269,3cyclictest0-21swapper/120:08:591
1332799261,9cyclictest0-21swapper/122:07:431
1332799261,9cyclictest0-21swapper/120:02:111
1332799261,12cyclictest0-21swapper/120:49:551
1332799261,12cyclictest0-21swapper/119:33:431
1332799261,11cyclictest0-21swapper/121:11:281
1333899251,12cyclictest12281-21/usr/sbin/munin00:00:123
1332799258,7cyclictest0-21swapper/122:17:471
1332799257,3cyclictest20197-21users19:20:271
1332799253,13cyclictest869-21sssd_nss19:35:191
13327992514,3cyclictest24875-21users00:25:241
1332799251,15cyclictest23-21ksoftirqd/120:23:271
1332799251,12cyclictest18092-21ntp_states20:30:191
1332799251,10cyclictest0-21swapper/100:34:391
1332799250,5cyclictest0-21swapper/122:10:271
1332199257,15cyclictest8632-21awk21:25:010
1332199257,15cyclictest8632-21awk21:25:010
1332199252,11cyclictest5470-21awk23:45:010
1333899241,16cyclictest0-21swapper/300:05:123
1333899240,17cyclictest0-21swapper/321:32:463
1333899240,15cyclictest0-21swapper/319:42:223
1333499246,8cyclictest0-21swapper/200:00:422
1332799249,3cyclictest16748-21df19:15:121
1332799247,3cyclictest0-21swapper/100:22:071
1332799241,9cyclictest0-21swapper/121:03:161
1332799241,10cyclictest0-21swapper/123:13:591
1332799241,10cyclictest0-21swapper/121:23:081
1332799241,10cyclictest0-21swapper/121:23:071
1332799241,10cyclictest0-21swapper/119:57:471
1332799240,9cyclictest0-21swapper/100:08:471
1333899231,15cyclictest0-21swapper/300:33:463
1332799237,3cyclictest0-21swapper/122:50:351
1332799236,4cyclictest0-21swapper/121:19:201
1332799236,3cyclictest0-21swapper/121:46:471
1332799231,9cyclictest0-21swapper/120:43:031
1332799231,8cyclictest0-21swapper/120:18:431
1332799231,19cyclictest2030-21diskmemload21:40:011
1333899227,6cyclictest0-21swapper/319:35:183
1333899222,11cyclictest2871-21/usr/sbin/munin22:25:163
1333899221,11cyclictest0-21swapper/323:14:433
1333899221,10cyclictest0-21swapper/319:27:403
13338992210,3cyclictest4396-21memory23:40:173
1332799228,4cyclictest30619-21nfs4_client19:45:191
1332799227,2cyclictest2030-21diskmemload22:56:151
1332799226,6cyclictest0-21swapper/122:34:471
1332799226,3cyclictest0-21swapper/122:36:311
1332799226,3cyclictest0-21swapper/121:51:191
1332799225,4cyclictest0-21swapper/100:16:031
1332799221,9cyclictest0-21swapper/120:53:471
1332799221,18cyclictest11046-21users22:40:261
1332799221,15cyclictest0-21swapper/123:20:031
1332799221,15cyclictest0-21swapper/123:00:071
1332799221,15cyclictest0-21swapper/122:27:571
1332799221,10cyclictest21171-21forks19:25:151
1332199221,11cyclictest15236-21perf21:40:010
1333899214,11cyclictest800-21snmpd00:27:393
1333899211,9cyclictest0-21swapper/323:30:463
1333899211,9cyclictest0-21swapper/322:05:143
1333899211,7cyclictest17836-21awk20:30:163
1333899211,10cyclictest0-21swapper/322:30:403
1333499215,9cyclictest31675-21unixbench-2d21:00:242
1332799216,8cyclictest0-21swapper/121:05:281
1332799215,3cyclictest0-21swapper/123:31:521
1332799215,3cyclictest0-21swapper/122:23:191
1332799211,9cyclictest0-21swapper/123:17:551
1332799211,15cyclictest0-21swapper/121:40:551
1332799211,15cyclictest0-21swapper/121:30:201
1332799211,14cyclictest0-21swapper/123:55:021
1332799211,10cyclictest12712-21/usr/sbin/munin22:45:191
1332799211,10cyclictest0-21swapper/123:40:361
1332799210,16cyclictest0-21swapper/100:39:081
1332799210,15cyclictest0-21swapper/120:57:071
1332199211,2cyclictest28615-21grep20:55:150
34932201,15sleep33527-21swap21:10:223
32952201,3sleep00-21swapper/019:56:150
18922200,4sleep232-21ksoftirqd/222:20:232
1333899207,8cyclictest0-21swapper/300:39:403
1333899204,6cyclictest21423-21/usr/sbin/munin20:40:223
1333899203,6cyclictest39-21rcuc/322:15:133
1333899201,9cyclictest0-21swapper/319:21:453
13338992013,4cyclictest23719-21grep20:45:113
13338992012,6cyclictest11271-21kworker/u8:019:32:493
1333899201,12cyclictest0-21swapper/321:55:243
1333899201,11cyclictest0-21swapper/322:20:243
1333899201,10cyclictest8846-21/usr/sbin/munin21:25:173
1333899201,10cyclictest0-21swapper/323:18:463
1333899201,10cyclictest0-21swapper/321:04:413
1333899201,10cyclictest0-21swapper/300:21:413
1333499205,7cyclictest2030-21diskmemload22:48:262
1333499205,7cyclictest13540-21grep19:10:082
1333499201,9cyclictest0-21swapper/222:25:102
13334992014,3cyclictest21454-21cut19:25:162
13334992010,3cyclictest20255-21sed00:15:242
1332799207,3cyclictest0-21swapper/119:10:151
1332799206,3cyclictest0-21swapper/100:11:551
1332799201,3cyclictest9163-21sed23:50:241
1332799201,14cyclictest0-21swapper/123:39:321
13327992011,2cyclictest2030-21diskmemload23:26:211
1332799201,11cyclictest12613-21expr00:00:121
1332199202,6cyclictest12047-21munin-run00:00:010
1332199202,10cyclictest22037-21/usr/sbin/munin21:55:130
1332199202,10cyclictest21147-21sshd20:39:330
1332199201,13cyclictest0-21swapper/022:54:330
1332199201,11cyclictest32416-21/usr/sbin/munin21:05:110
265942191,3sleep00-21swapper/023:17:130
18812191,3sleep20-21swapper/221:08:212
1333899198,3cyclictest7875-21/usr/sbin/munin23:50:253
1333899196,3cyclictest0-21swapper/319:58:393
1333899196,11cyclictest7923-21switchtime21:20:243
1333899196,11cyclictest7923-21switchtime21:20:243
1333899193,7cyclictest20464-21/usr/sbin/munin23:05:193
1333899191,8cyclictest0-21swapper/320:15:393
1333899191,7cyclictest8230-21/usr/sbin/munin20:10:253
1333899191,7cyclictest8230-21/usr/sbin/munin20:10:253
1333899191,4cyclictest19226-21/usr/sbin/munin20:35:163
13338991914,3cyclictest26283-21grep20:50:153
1333899191,10cyclictest4882-21ntpq20:00:193
1333499196,3cyclictest1822-21expr19:55:122
1333499195,4cyclictest8230-21/usr/sbin/munin20:10:162
1333499195,4cyclictest8230-21/usr/sbin/munin20:10:162
1333499194,7cyclictest29804-21ps22:10:232
1333499191,5cyclictest821-21lldpd21:27:002
13334991914,3cyclictest1399-21turbostat.cron19:55:002
1333499191,2cyclictest24428-21sshd23:12:352
13334991912,5cyclictest13309-21cyclictest00:27:392
1332199194,9cyclictest0-21swapper/021:10:200
1332199193,3cyclictest87350irq/128-enp1s0-22:40:130
13321991912,4cyclictest10814-21dump-pmu-power21:30:010
1332199191,12cyclictest0-21swapper/021:00:230
1332199191,12cyclictest0-21swapper/000:25:180
1332199191,10cyclictest0-21swapper/019:20:260
1332199190,10cyclictest29851-21grep23:25:170
210902181,2sleep00-21swapper/000:20:090
1333899187,2cyclictest0-21swapper/323:22:023
1333899185,3cyclictest87150irq/126-enp1s0-22:10:153
1333899183,7cyclictest0-21swapper/300:19:333
1333899182,6cyclictest8007-21munin-run20:10:013
1333899182,14cyclictest0-21swapper/323:25:053
1333899181,4cyclictest19840-21/usr/sbin/munin21:50:153
1333899181,3cyclictest18404-21if_enp2s021:45:163
1333899181,3cyclictest0-21swapper/321:15:223
1333899181,2cyclictest16478-21cpuspeed_turbos19:15:103
1333899181,15cyclictest0-21swapper/323:58:303
1333899180,7cyclictest0-21swapper/320:58:033
1333499187,3cyclictest87450irq/129-enp1s0-20:00:192
1333499185,3cyclictest831-21runrttasks21:52:332
1333499185,3cyclictest19496-21sort19:20:212
1333499185,3cyclictest16703-21/usr/sbin/munin00:10:112
1333499185,3cyclictest0-21swapper/223:53:102
1333499184,7cyclictest21188-21tr00:20:132
1333499184,7cyclictest20191-21sshd23:04:342
1333499184,4cyclictest15444-21/usr/sbin/munin21:40:172
1333499184,3cyclictest25850-21needreboot23:15:192
1333499183,8cyclictest31592-21if_enp2s022:15:172
1333499183,3cyclictest0-21swapper/220:55:262
1333499181,5cyclictest3483-21/usr/sbin/munin23:40:142
1333499181,4cyclictest26574-21ethtool20:50:172
1333499181,4cyclictest0-21swapper/220:35:192
13334991812,3cyclictest7840-21latency_hist23:50:012
1333499181,15cyclictest0-21swapper/221:55:062
1333499181,10cyclictest0-21swapper/200:30:182
13334991810,3cyclictest25207-21cat22:00:192
1333499180,5cyclictest0-21swapper/222:30:272
1332199187,3cyclictest23616-21/usr/sbin/munin20:45:210
1332199186,4cyclictest8322-21ntpq22:35:220
1332199183,8cyclictest87350irq/128-enp1s0-19:40:480
1332199183,8cyclictest17332-21kworker/0:121:46:590
1332199181,9cyclictest18261-21/usr/sbin/munin23:00:130
1332199181,11cyclictest0-21swapper/019:15:160
1332199180,9cyclictest16588-21pmu-power21:40:210
290292171,2sleep20-21swapper/219:40:482
275782170,2sleep00-21swapper/020:52:330
231722171,3sleep00-21swapper/020:42:230
231022171,3sleep20-21swapper/220:41:282
222032170,2sleep00-21swapper/023:07:140
172792170,2sleep30-21swapper/321:43:253
1333899179,2cyclictest87150irq/126-enp1s0-22:48:293
1333899174,4cyclictest15069-21expr20:25:133
1333899173,3cyclictest1431-21timerandwakeup21:05:243
1333899173,3cyclictest0-21swapper/323:49:433
1333899173,3cyclictest0-21swapper/323:49:433
1333899171,6cyclictest13853-21sh19:10:093
1333899171,5cyclictest0-21swapper/300:10:153
13338991713,3cyclictest0-21swapper/322:38:303
13338991713,3cyclictest0-21swapper/319:47:503
1333899171,2cyclictest16885-21kernelversion22:55:173
13338991712,3cyclictest24762-21fschecks_time22:00:163
1333899171,14cyclictest0-21swapper/321:35:273
1333899171,13cyclictest564-21find19:50:243
1333899171,13cyclictest12623-21/usr/sbin/munin20:20:173
1333899171,10cyclictest0-21swapper/323:35:403
1333899171,10cyclictest0-21swapper/322:42:393
1333499176,4cyclictest15826-21cut20:25:202
1333499175,3cyclictest6036-21/usr/sbin/munin20:05:162
1333499174,3cyclictest30548-21cat23:25:252
1333499174,3cyclictest29786-21cstates19:45:122
1333499173,7cyclictest7119-21fschecks_time21:20:152
1333499173,7cyclictest7119-21fschecks_time21:20:152
1333499171,9cyclictest14604-21munin-run20:25:012
1333499171,7cyclictest25203-21/usr/sbin/munin19:35:262
1333499171,6cyclictest27051-21/usr/sbin/munin23:20:182
1333499171,6cyclictest24769-21sshd19:32:322
1333499171,6cyclictest10428-21acpi20:15:112
1333499171,3cyclictest0-21swapper/221:30:132
1333499171,3cyclictest0-21swapper/221:15:122
1333499171,3cyclictest0-21swapper/221:14:222
1333499171,2cyclictest0-21swapper/222:05:242
1333499171,14cyclictest0-21swapper/222:51:362
13334991711,3cyclictest7197-21tail23:45:272
13334991711,3cyclictest31473-21cpu23:30:122
13334991711,3cyclictest17813-21timerwakeupswit19:15:222
1332199175,3cyclictest30763-21ps19:45:220
1332199174,7cyclictest29172-21kworker/0:219:25:290
1332199173,8cyclictest87350irq/128-enp1s0-20:25:200
1332199173,12cyclictest25494-21/usr/sbin/munin00:30:230
1332199172,12cyclictest30835-21/usr/sbin/munin22:15:230
1332199171,6cyclictest14473-21sshd20:23:340
1332199171,6cyclictest0-21swapper/000:00:240
1332199171,5cyclictest29658-21munin-run00:40:000
1332199171,5cyclictest2030-21diskmemload22:55:230
1332199171,2cyclictest17631-21grep20:30:150
13321991712,3cyclictest28638-21/usr/sbin/munin22:10:210
13321991712,3cyclictest19840-21/usr/sbin/munin21:50:220
13321991712,3cyclictest10979-21perl20:15:160
1332199171,13cyclictest31448-21/usr/sbin/munin23:30:260
1332199171,13cyclictest1284-21/usr/sbin/munin23:35:120
261172160,2sleep00-21swapper/019:35:200
248062160,2sleep00-21swapper/019:32:590
13338991610,4cyclictest821-21lldpd23:03:023
1333499163,3cyclictest11208-21chrt22:42:182
1333499161,5cyclictest17101-21sort20:30:112
1333499161,5cyclictest16146-21cpuspeed_turbos22:55:122
1333499161,4cyclictest14843-21diskstats00:05:162
1333499161,4cyclictest0-21swapper/223:05:472
13334991611,3cyclictest28815-21pmu-power00:35:252
13334991611,3cyclictest23616-21/usr/sbin/munin20:45:122
13334991611,3cyclictest13244-21/usr/sbin/munin21:35:162
1332199169,4cyclictest16768-21cpuspeed_turbos00:10:110
1332199169,4cyclictest0-21swapper/019:50:190
1332199162,8cyclictest87150irq/126-enp1s0-22:20:260
1332199161,7cyclictest0-21swapper/023:50:290
1332199161,6cyclictest6036-21/usr/sbin/munin20:05:200
1332199161,6cyclictest3949-21cpuspeed_turbos20:00:110
1332199161,6cyclictest24768-21fschecks_time22:00:150
1332199161,3cyclictest27051-21/usr/sbin/munin23:20:210
13321991611,3cyclictest5074-21/usr/sbin/munin22:30:180
13321991611,3cyclictest12603-21missed_timers22:45:190
1332199161,12cyclictest5685-21/usr/sbin/munin23:45:250
1332199161,12cyclictest5685-21/usr/sbin/munin23:45:250
1332199161,12cyclictest18902-21/usr/sbin/munin00:15:170
13321991610,3cyclictest3211-21sh22:25:120
1333899150,6cyclictest0-21swapper/322:50:143
1333499151,4cyclictest17643-21/usr/sbin/munin21:45:132
1332199158,4cyclictest26995-21tune2fs22:05:160
1332199151,5cyclictest14686-21ps19:10:190
1332199151,4cyclictest800-21snmpd00:05:060
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional